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chool performance frameworks (SPFs) are action-oriented information tools 

that provide information on school performance and quality across a variety 

of measures. SPFs have evolved as a solution for local communities to make 
sense of an array of student, school, and system performance data, and to support 
transparent and equitable decisions. SPFs can be especially useful to communities 
grappling with various autonomous public school structures, school accountability 


systems, broader school choice, and ambitious school improvement goals. 


SPFs are often talked about as report cards or progress reports, but they are more 
than that. What sets an SPF apart from other data systems is the fact that it is 
explicitly designed to inform action that can lead to greater student achievement 
— but the actions an SPF might support, and the design choices behind them, 
vary considerably. Some, but not all, SPFs include a composite rating or score 

of school performance such as a letter grade or tier. Others present a range of 
data as a “dashboard” that may include growth and performance results from 
standardized tests, indicators of achievement gaps, indicators of school climate 


and environment, and other factors of particular interest to different stakeholders. 


Creating a tool that 
can serve school 
leaders alongside 
families and system 
leaders requires some 
creative thinking, 
and could require 
separate, but aligned, 
ways of looking at 
performance data. 


Different local users might need an SPF for different purposes. Key user groups 
include system leaders, school leaders, families, educators, and community 
members at large, and their needs and preferences may diverge significantly. 
System leaders, school leaders, and families should have shared information at 
their disposal and a shared understanding of what defines a high-quality school. 
Transparency and equal access to information are hallmarks of an equitable 
system. But because system leaders are generally the people designing and 
operating SPFs, many SPFs focus on district leaders’ needs, and could be more 
transparent or useful to the public. Over time, some SPFs have evolved to be 


more intentionally school- and public-facing. 


In order to optimize an SPF for a particular goal or purpose, leaders may need 

to make tradeoffs. System leaders should understand those tradeoffs and make 
intentional design choices that reflect their goals. Too often, system leaders learn 
late in the process that results from an SPF are being used in unanticipated ways. In 
particular, the level of detail and kinds of data school leaders need in order to inform 
ongoing decisions diverges from the higher level of detail and focus on outcomes 
that system leaders and families prefer. Creating a tool that can serve school leaders 
alongside families and system leaders requires some creative thinking, and could 


require separate, but aligned, ways of looking at performance data. 


As SPFs have grown in use and popularity at the local level, the research base 
supporting them has remained thin, and is now largely outdated. There has been 
surprisingly little written about the various long-standing SPFs, how they have 
evolved over time, how they compare with each other, and the lessons they offer 


for other communities. 


In recent years, much of the conversation around how to assess school 
performance has focused on states’ Every Student Succeeds Act (ESSA) reporting 
and accountability systems, and the pros and cons of various metrics, methods, 
or rating schemes that states might use. Conversations about state accountability 
systems are valuable, but creating and applying an SPF at the local level entails 


different goals and considerations than a statewide system required by federal law. 
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In an ideal scenario, local SPFs add nuance and a focus on local priorities and 
context to a state rating and accountability system — state accountability systems 
and local SPFs should not confuse families or present opposing narratives. They 
also are designed to support different actions at a local level than a state system, 


and may have a higher level of local credibility. 


Now is the right time to learn from existing SPFs that have been in place at a local 
level over multiple years, and draw lessons to build stronger systems designed for 


the conditions and strategic questions school systems face today. 


This report is intended primarily for local policymakers and leaders in school 
systems considering a new SPF or revising an old one. We hope that this will 

be an informative and relevant deep dive on a topic that is top of mind for many 
system leaders. Other readers, including state educational leaders, advocates, 
parents, or anyone interested in how school systems define school performance 


and quality, may find this report useful. 


This report can be read top to bottom, or by jumping around to sections that are of 


most interest. The report is organized as follows: 


e Background and Research: An overview of key context and research shaping 


the way SPFs are used today 


e¢ Defining Use Cases: An explanation of primary use cases that shape SPFs’ 
form and function, and form the backbone of our analysis. “Use case” is a term 
borrowed from the world of technology and software design.’ Designers create 
use cases to describe in detail how a system or product might be used — we 
argue that leaders designing SPFs would benefit from a similar mindset: 
yr)" re 
vo" System Management and Accountability: System leaders (school 
“~ districts, states, charter school authorizers) can use SPFs to help 
make system management and accountability decisions about 
school expansion, recognition, renewal, sanction, investigation, 
and closure. SPFs can help identify struggling schools in need of 


support and exemplar schools worth replicating. 


School Continuous Improvement: An SPF can help school leaders 
identify and address areas for improvement, and make strategic 
choices in day-to-day management. This is separate from system- 
wide continuous improvement, and is primarily meant to serve school 
leaders. SPFs intended to support school continuous improvement 
use data to help school leaders make decisions about action plans, 
interventions, and resource allocation in order to improve student 


performance and school quality on an ongoing basis. 


Family and Community Information: Families and community 
members can use SPFs to understand their school’s performance, 
navigate the school choice process, and advocate for 
improvement. Cities with a high degree of school choice often 
turn to an SPF to provide reliable, clear, public-facing information 
about school performance. This information can be the linchpin 
of a school choice system that empowers families and levels the 


informational playing field of the school choice process. 


Applying Use Cases to Existing SPFs: Brief summaries of five established, 


locally focused SPFs and how their intentions and design serves each use 


case. These SPFs are not necessarily exemplars. Each has different strengths 


and weaknesses, and levels of alignment to the three use cases. System 


leaders with different goals can learn different things from each of these SPFs: 


1. Washington, D.C.: Public Charter School Board, Performance Management 


Framework 


a -F WN 


New York City: NYC Department of Education, School Quality Reports 
Chicago: Chicago Public Schools, School Quality Rating Policy 
Denver: Denver Public Schools, School Performance Framework 


New Orleans: Louisiana Department of Education and Orleans Parish 


School Board, Louisiana School Performance Scores 


Lessons From Existing SPFs: Based on analyses of the five systems above, 


answers to some of the questions local leaders may have about how to 


design, use, and sustain an SPF, what lessons can be learned from the 


established SPFs above, and how SPFs should align with local needs and 


goals for student and school success. 
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e SPF Planning Guide: Takeaway considerations for local leaders considering 


an SPF, including a three-step process toward SPF creation. 


e Detailed SPF Profiles: Full profiles and analyses of the five featured SPFs, 
including metrics, methods, evolution, design considerations, and detailed 


explanation of alignment with the three primary use cases. 


Building a Continuously Improving System of Schools 


Many communities that create SPFs do so as part of larger improvement strategies. All five of the SPFs 
highlighted in this report are also featured in “Eight Cities,” a 2018 Bellwether publication at 
www.EightCities.org that tells the story of urban school systems that have seen student outcomes improve 
at a faster pace than other urban school systems. An SPF is one of seven strategic pillars Eight Cities’ 


authors identify as key to improvement: 
y PW oX=J aco) guit-larex-mie-ln(=\1'(e) a a] 9) 9) |(=ssmce la] | 016] 0) | (omsyel alele) [S 
2. School-level autonomy around staffing, budgetary, and instructional decisions 
3. A performance contract between schools and an oversight body that monitors performance 


4. One or more school quality oversight bodies that make opening and closing decisions based on 


oi akoXo) me [Ure] |inyaxexe)anlaalelaviNvalar=t=xermrs] aCeM col aalihare(=saatslare! 
Way -Na oy ti eX= Mo) mre) cel= lal y4al ilo) miorel else Melam=>¢er-]alellaleAma-ye)[cetzluiare palace lavel ll ey=1 Uae mal-wmcte alete)|s 
6. A talent strategy for teachers and principals 
7. A unified enrollment system across all public schools 


Leaders in each city in the report created a unique approach to implementing these strategic pillars catered 
to their local context, yet they often began their efforts with the creation of an SPF. SPFs created a common 
definition for school performance that enabled the implementation of other strategic pillars such as 

expanding, replicating, and incubating new schools. For more on the featured cities and their improvement 


stories, visit www.EightCities.org. 
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Background and Research 


PFs are a relatively new tool in many places and well established in 

others. Several of the local SPFs in use today were developed, in part, 

as a reaction to the federal No Child Left Behind Act (NCLB, 2001) 
school ratings. NCLB established national requirements for standards-based 
accountability and relied primarily on state assessment proficiency in math 
and reading to measure school performance. Local SPFs arose in part as a 
supplement, or in some cases an alternative, to NCLB ratings. Local systems could 
be more responsive to local goals, and incorporate other measures of school 
performance such as student growth metrics and more qualitative measures of 


school environments. 


The Every Student Succeeds Act (ESSA, 2015), replaced NCLB and required 
revisions to state report cards. Most states chose to pursue a wider range of 


metrics than what was allowable under NCLB.? 


In parallel to these state and federal developments, a growing school choice 
movement? and increased adoption of school governance structures emphasizing 
accountability and autonomy’ means that more communities want clear standards 
for local school accountability and public-facing information about school 


performance.°® 
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SPFs are first and 
foremost a tool that 
supports policies 

and practices that 
can drive improved 
outcomes for students 
at the family, school, 
and/or systems level. 
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The theory of action behind SPFs has direct ties to research on data-driven 
decision-making in education.® This research suggests that effective use of 

data — about students, achievement, and schools — is associated with stronger 
outcomes, but only when the users of data are able to make sense of it.” Raw 
scores or spreadsheets are not sufficient to analyze and synthesize outcomes 
data and provide a strong basis for important decisions like what schools to open 
or close, what school to choose for your child, or what programs to implement 

in a school building. SPFs aim to provide a clear, reliable framework at a school 
level for understanding school performance, and making strong data-informed 


decisions. 


The most directly relevant research focused specifically on SPFs comes from 

the Center for Reinventing Public Education (CRPE). CRPE’s “Apples to Apples” 
(2016) project focused on SPFs spanning charter and district schools.’ The report 
explains how leaders can navigate process and buy-in considerations when 
collaborating to create a tool that serves multiple kinds of schools, recommends 
SPF metrics be aligned to clear goals, and advises a gradual, measured 
implementation plan. Beyond CRPE’s work, information on best practices in 


process, policy, and goals for local SPFs is scant. 


SPFs are first and foremost a tool that supports policies and practices that can 
drive improved outcomes for students at the family, school, and/or systems level. 
The relationship between SPFs and student outcomes is indirect, and a clear 
causal link between the SPF tool and improved student achievement is elusive. 
Although it was not part of our selection process, all the systems highlighted in 
this report have seen improved student outcomes in years that coincide with 

the SPF, along with other consequential reforms.? But direct impacts on student 
achievement is likely the wrong measure of success for an SPF — rather, SPFs 
should be measured on their ability to reliably and effectively support system 
leaders, families, and/or school leaders in making a variety of important decisions 


that in turn directly affect students. 


An adjacent body of research focuses on the impact of state-level performance 
ratings and accountability more generally, especially under NCLB. For example, 
early research on school accountability found positive benefits for students 

in states with strong accountability systems, where strength is defined as the 


application of clear consequences for schools based on student achievement.” 


Other studies have found that reporting data alone is not enough to drive 
school improvement. Schools needed the additional motivation of consequential 


interventions to prompt action and improve results." 


Another body of relevant research focuses specifically on families’ understanding 
of and interaction with school data. If data about school performance is presented 
to families in different ways, they will behave differently when choosing a school. 
Researchers found that when schools were organized on a website by academic 
performance instead of alphabetically, parents were more likely to choose schools 


with higher academic performance. 


But even in places where SPFs are designed to be more family-friendly, it is 
unclear how many families are aware of or use these kinds of data sources 

and reports. Student report cards and discussions with other families, teachers, 
and principals tend to be parents’ primary sources of information for how their 
student and school are doing.” State or district report card websites can be 
invaluable additional sources of information for families, if they are designed and 
disseminated with families’ needs in mind. Even in cities like New Orleans with a 
high level of school choice and without traditional neighborhood schools,’ local 
organizations report that families have mixed understanding and awareness of 


school performance grades and what they mean. 


Families often face substantial barriers to accessing or using school quality reports, 
especially if they do not have easy internet access or primarily speak a language 
other than English.” In discussions with advocacy groups working directly with 
families, many thought that where SPFs existed, they were at best inadequate to 


family needs, and at worst, intentionally obscured information for families. 


We found several examples in our research of school report card websites 

that were filled with jargon and only available in English. Family surveys and 
focus groups conducted by Learning Heroes and the National PTA indicate 

that terminology that is commonplace among education insiders may mislead 
families."® For example, a parent might think that “student growth” means growth 
in school enrollment, rather than academic improvement, or “climate” means 
whether the school has air conditioning, rather than indicators like suspension/ 


expulsion or student and teacher satisfaction.® 
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An SPF provides 
an opportunity for 


communities to define 


what matters to them. 
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Without an SPF, school quality tends to be defined by word of mouth, high-level 
test scores, and/or measures like neighborhood income. An SPF provides an 
opportunity for communities to define what matters to them. Third-party data 
sources created by nonprofits or businesses may step in to fill the void when 
a district is unwilling or unable to meet family needs for information, but their 


information may not be the most comprehensive, unbiased, or up-to-date. 


This underlines the importance of purposeful consideration of different use cases 
for SPFs. If an SPF is meant primarily to serve families, it should be designed with 
their needs in mind. If an SPF is designed to serve school leaders first, the design 
will likely be different. In other words, the form of an SPF should directly follow 
from its function, and the users on the receiving end of SPF results should directly 
shape that form. 


DYaureveets | OKon Or as 


We have identified three primary purposes for which 
SPFs can be used and designed: 


vr ¢ @ 


System Management School Continuous Family and Community 
and Accountability Improvement Information 


The concept of a “use case” is adopted from the world of technology and software 
design. A use case provides a structured way for designers to envision how their 
product will be put to use and align design decisions to end users’ needs.”° In 

the case of SPF design, system leaders would benefit from early considerations 

of what goals and users they hope to serve, and what use cases they want to 


prioritize for their SPFs. 
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SPFs exist in a broader 
school ecosystem that 
includes other supports 
and sources of data — 
not every SPF needs 

to serve all use cases 

in the same way, or 

to the same degree, 
especially if there 

are other supports 


available. 
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Each of these use cases involves different users and priorities for what an SPF 
could include or accomplish. SPFs exist in a broader school ecosystem that 
includes other supports and sources of data — not every SPF needs to serve all 
use cases in the same way, or to the same degree, especially if there are other 
supports available. But leaders should be clear and intentional about which use 


cases they are trying to serve, or not. 


The most common use case is System Management and Accountability; however, 
the other two purposes also exist in the field. SPFs designed for use as a School 
Continuous Improvement tool are the least common. And while none of the 
systems we examined was initially designed primarily to provide Family and 
Community Information, in several cases, over time several SPFs recognized 


families as a user group and adjusted some aspects of their systems in response. 


Across these three use cases, any successful SPF needs to start from a similar 


foundation of resources and information: 


e Reliable, comparable data. Data on key school and student success metrics 
across all schools in the system should be collected on a regular basis. 
Where such data do not yet exist, the system needs the authority and ability 


to collect them. 


e Resources, in terms of budget, time, and staff capacity. Given the important 
decisions that SPFs drive and inform, system leaders must adequately 
support both the creation of an SPF and its implementation to ensure a high 


degree of quality. 


e Stakeholder alignment. An SPF needs alignment (or the potential for 
alignment) among key stakeholders on the purpose and goals for the system 
to be successful. A technically “perfect” system that lacks buy-in from key 
stakeholders will not gain the trust of potential users, undermining its ability 
to support key decisions and its durability over time. Building buy-in requires 
authentic engagement across the users and stakeholders for a system, 


especially school leaders and families. 


GOALS 


e Intentional Communication. At its core, an SPF is itself a tool for 
communicating information about schools to stakeholders. In order for it to be 
successful, system leaders must consider how those stakeholders will receive 
information both about the SPF and from it. Absent a strong communications 


strategy, an SPF risks losing credibility and impact with critical audiences. 


The use cases below explain the users, goals, and activities that should drive 
an effective SPF within each case. We then apply the use cases as an analytical 
tool for assessing the strengths and weaknesses of different SPFs with different 
goals in mind. These use cases were created by our team based on a synthesis 
of available SPF-related literature, interviews with SPF leaders and experts in 
the field, and prior work at Bellwether on components of effective and equitable 
school accountability systems. 


System Management and Accountability 


System leaders, such as authorizers, school board members, or superintendents, 


making decisions around school management and system-wide strategies. 


System leaders may have a variety of short-term goals that an SPF could support, 


including: 
e Identifying schools in need of intervention or intensive supports 


e Identifying and rewarding high-performing schools, identifying and 
disseminating best practices 


e Making decisions around school expansion, replication, or closure 


e Designing system-wide improvement strategies and allocating resources 


appropriately and equitably in line with those strategies 


All these decisions should be made on a fair, consistent, and transparent basis, 
consistent with long-term goals and the overarching strategy for the system. If system 
leaders have a clear sense of performance across schools, then system resources and 
management decisions will be aligned to goals and priorities for student outcomes, 
ideally resulting in a system with higher-performing schools, a greater number of high- 


quality seats for students, and improved student outcomes across student subgroups. 
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She most important activities in creating an SPF to serve system leader goals are: 


1. 


Select valid and reliable metrics that align with long-term system goals for 

students. 

a. Use growth/progress measures as well as overall measures of student 
outcomes. 

b. Disaggregate student data to identify and monitor performance gaps 
and growth trajectories, especially for historically underserved student 


subgroups. 


. Combine weight and benchmark metrics in order to reflect the relative 


priority of each metric and produce differentiated results among schools that 
reflects their relative performance. 
a. Create a system that can account for programmatic differences across 


schools (e.g., grade levels, innovation plans). 


. Present data in clear and understandable ways that answer questions system 


leaders may have about school performance. 


. Design and implement clear policies and processes for how results will inform 


system decisions — what will the system leadership do with this information? 
What will it mean when a school is performing well, or not performing well, on 


the SPF metrics? 


. Monitor performance on a regular basis, with more intensive monitoring for 


lower-performing schools or schools where data flag potential problems. 


. Communicate decision-making priorities, processes, and standards clearly to 


school leaders and community members on a proactive, ongoing basis. 


¢ School Continuous Improvement 


USERS School leaders (e.g., principals, academic leaders) and supporting system staff 


(e.g., specialized district support staff). 


GOALS School leaders want reliable and clear data that can inform and support 
continuous improvement in their schools’ operations, culture, and student 


outcomes. Their uses for an SPF might include: 


e Understanding how their school is performing against district/system 


expectations and goals 


e Diagnosing key strengths and weaknesses across a variety of outcomes and 


flagging leading indicators of potential problems for early action 


e Guiding school resource allocations, staffing plans, and programmatic 


decision-making 


In contrast to the system management use case, school-level continuous 
improvement is not focused on relative performance across schools. Depending 
on the school and the system, school leaders might have different levels of 
control over things like staffing and budget. In cases where school leaders have 
relatively high autonomy, a granular and reliable understanding of their school’s 


performance is even more important. 


Information from an SPF oriented toward school continuous improvement should 
enhance school leaders’ ability to diagnose and address problems and use data 
to drive improvement in their buildings. If that occurs, in the longer term, schools 
should meet or exceed system expectations for student outcomes, and achieve a 


high-functioning school environment for students and staff alike. 
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She key activities in creating an SPF to serve school continuous improvement are: 


1. 


oOo un BF W 


Select an array of valid and reliable metrics, including long-term outcomes 
for students; interim, diagnostic, or benchmark results; and leading 


indicators of success, which are all actionable at the school level. 


a. Consider differentiated metrics aligned with program goals (e.g., early 


learning measures for a PreK-3 school). 


b. Disaggregate student data to identify and monitor performance gaps 
and growth trajectories, especially for historically underserved student 


subgroups. 


. Benchmark school metrics against appropriate and useful comparisons or 


research-based goals. 


. Engage school leadership teams as key stakeholders in the design process. 
. Flag strengths and weaknesses on an ongoing basis. 
. Share data with school leaders in usable, clear formats that fit their needs. 


. Incorporate results into school continuous improvement processes, planning, 


and decision-making throughout the year. 


. Train school leadership teams in how to use the tool effectively and link it to 


day-to-day decisions. 


. Communicate data and ongoing action plans to staff, families, and 


community members. 


Family and Community Information 


Families and community members 


An SPF serving family and community needs should provide an equitable and 


transparent opportunity for families from different backgrounds to understand 


how their schools are performing, make choices for their children, and advocate 


for change. Families and community members could have several goals for SPF 


information, including: 


Reliable, relevant, and useful information to inform school choices, where 


those options exist 
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Data to understand their schools’ performance in order to place their students’ 
current performance in context (e.g., interpret report cards and test scores, 


advocate for services or programmatic changes at the school or system level) 


Information that can help them effectively engage with the school system, 


advocate for community priorities, and choose elected leaders 


If families and community members have this kind of information from an SPF, 
system leaders may be more responsive to community priorities, families and 
students will experience higher levels of satisfaction and engagement with 


schools, and demand will rise for higher-performing schools and systems. 


The key activities in creating an SPF to serve families and community members 


1. Engage families and community members to understand their questions and 


priorities and shape system design around the users. 


. Select metrics that align with system priorities, in addition to information 


families and community members most want to know: 

a. How are students performing? 

b. How are students progressing? 

c. What is the experience of families and students at the school? 


d. What programs or amenities does the school offer? 


. Translate performance on those metrics, in subcategories of performance 


and on performance overall, using relevant comparisons and/or easy-to- 
understand graphs, labels, or ratings. 
a. Present the most important information up front, at a glance. 


b. Allow for deeper dives to understand what each rating and metric means. 


. Actively communicate results to families and community members in a variety 


of accessible ways. 

a. Present data to families in a manner that is accessible and free of jargon. 
Use accessible language and provide high-quality multilingual translations, 
context, and definitions. 

b. Provide families with multiple opportunities to access, understand, and use 
the data, and discuss it with teachers, school leaders, and system leaders. 

c. Reach families where they are through outreach and engagement online 


and in person — don’t rely on families to find your SPF on their own. 
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Applying Use Cases to Existing SPFs 


he five systems we chose to examine represent a spectrum of SPF 

approaches in terms of who controls the SPF, the SPF’s explicit and 

implicit goals, the users it serves, metrics and ratings, and communications 
strategies. The aim of this analysis is not to rate or grade these five systems 
against one another, or present any of the five as exemplars. Rather, it is to 
provide real-world examples of how SPFs today operate relative to their intended 
purposes as well as other potential purposes and how SPFs in the future might 


learn from these experiences. 


This section provides summary information about each of the five SPFs as relates 
to each of the three use cases. For more information, detailed profiles begin 

on page 28. Analyses are based on scans of publicly available documentation, 
policies, and displays of SPF data current as of the 2018-19 school year; interviews 
with system leaders in each community; interviews with local advocates and 


former SPF leaders; and local media and external reports about each system. 


Key SPE Information 


City SPF Creator SPF Name # of Types of Schools Included 
Schools 
Included 
Chicago, IL Chicago Public School Quality 644 All district-run, charter, and 
Schools Rating Policy contract schools in Chicago 
(SQRP) 
Washington, DC DC Public Charter Performance 123 All DC public charter schools 
School Board Management 
Framework (PMF) 
New York, NY New York City School Quality 1,800 All district-run schools and 
Department of Reports district-authorized charter 
Education schools 
New Orleans, LA Louisiana School 85 All schools in New Orleans, 
Department of Performance almost all of which are public 
Education and Scores charter schools. The system 
NOLA Public also applies statewide 
Schools 
Denver, CO Denver Public School 162 All district-run schools, 
Schools Performance including innovation schools 
Framework (SPF) and all district-authorized 


public charter schools 


These five SPFs serve systems ranging from the country’s largest school district 


(New York City) to the country’s only school system that is nearly all charter 


schools (New Orleans). Some apply to all schools in the city or state; others apply 


to a subset of schools based on authorizers and management. One SPF is a state 


ESSA system adapted for use at the local level. 


In terms of metrics and ratings, some SPFs were quite straightforward. For 


example, Louisiana’s ratings for elementary schools are based 75% on state test 


performance, and 25% on state test growth. Other systems, such as Denver and 


New York City, use a wide array of growth and performance metrics derived from 


student assessments alongside school environment measures such as student 
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survey results, attendance, achievement gap closure, and ratings from on-site 
reviews. All but one system (New York City) assigns a Summative rating or grade to 
schools. As the summaries below explain, these differences in design are driven in 


large part by differences in city context, goals, and priorities. 


Chicago Public Schools — School Quality 
Rating Policy (SQRP) 


The SQRP is a five-point rating system that applies to all CPS schools including public 


charter schools. The current iteration of the SQRP has been in place since 2013. 


Elementary/middle school ratings emphasize academic growth, closing gaps 
for priority student groups such as English learners, test performance, and 
attendance. High school ratings emphasize graduation and postsecondary 


readiness and outcomes. 


System Management and Accountability 


The SQRP was primarily designed to serve system management purposes. Highly 
rated schools have more local governance power and autonomy under the 
oversight of local school councils; low-rated schools are subject to more intensive 
intervention and support. Through the district’s charter accountability policy, 
charter schools with poor academic performance may be at risk of revocation. The 
stability of the SQRP and the clarity of its system management are assets in terms 
of credibility for informing important decisions. However, despite the emphasis on 
growth in the ratings, correlations between SQRP ratings, school demographics, 
and school segregation have generated some criticism and prompted CPS to plan 


an upcoming equity review of the SQRP.”" 


School Continuous Improvement 


The SQRP is not designed or intended to serve as a continuous improvement tool 
for school leaders. SQRP metrics do not differentiate between diagnostic metrics 


and key outcomes, and many metrics, especially at the high school level, are only 


w 


available after the end of the year. But CPS has taken steps in recent years to help 
school leaders better use their SQRP results to inform school-level action, and 


prioritize areas for improvement. 


As the state of Illinois debuts its new ratings under ESSA, conflicts between SQRP 
ratings and state ratings have already emerged and may send mixed messages to 


school leaders. 


Family and Community Information 


Currently, SQRP serves as a tool to help families navigate school options. Over 
time, the district has added more family-facing resources and integrated SQRP 
ratings into different family tools. CPS’ online school profiles stand out positively 
for readability and clear design choices meant to address families’ priorities,?? 

but there are still places where family-friendly communication falls short.?? For 
example, the ratings scale is not intuitive (1+, 1, 2+, 2, and 3), and in recent debates 


over SQRP changes, parent input was highlighted as a missing element.” 


DC Public Charter School Board — 
Performance Management F ramework 


(PMF) 


The PMEF is a three-tiered system that applies to all public charter schools in the 
District of Columbia. It was created by D.C.’s sole charter school authorizer, the DC 
Public Charter School Board (DC PCSB). 


Elementary/middle school ratings emphasize growth and achievement on state 
tests, with extra weight for third-grade reading and eighth-grade math. High 
school ratings are also based on growth and achievement on state tests, plus 
postsecondary readiness metrics such as college acceptance rates. Nonacademic 
metrics for all grades include attendance and reenrollment. The system also 
includes additional untiered, stand-alone measures for early grades that align 

with schools’ charter agreements. These untiered measures do not factor into the 


overall score or tier for the school. 
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The PMF was primarily designed to support DC PCSB’s decisions as a charter 
school authorizer. Tier 1 schools are encouraged to expand and are subject to 
less monitoring; Tier 3 schools are subject to a high-stakes review process and 
additional monitoring, and may have their charters revoked. The relationship 
between the PMF results and charter renewal decisions varies by school: Many 
schools have made PMF results central to their charter goals, but other schools 


have unique individual charter goals. 


Much like CPS, there are concerns within the charter school sector that the 

PMF correlates too closely with school demographics, despite the emphasis 

on growth metrics, and may disadvantage schools serving high populations of 
at-risk students. As a result, DC PCSB is considering adjustments, especially the 
emphasis on third- and eighth-grade performance metrics, in the near future. DC 
PCSB reviews all the components of the PMF every year and makes adjustments 


to address concerns raised by school leaders, families, and others. 


¢ School Continuous Improvement 


DC PCSB believes school-level improvement decisions are not part of its role 

as an authorizer, and DC PCSB does not have authority to mandate that schools 
administer assessments or collect and report data beyond what the state requires. 
The PMF is not designed or intended to serve as a continuous improvement tool for 
schools, but it does provide information about school performance to school leaders 
that could lay the foundation for other improvement efforts and direct school leaders 
to focus on outcomes that are important for students. Ultimately, school leaders can 


align their goals and improvement plans to the PMF metrics, or not. 


Family and Community Information 
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The three tiers of the PMF provide families with a simple way to differentiate 
among D.C. public charter schools. PMF reports for each school are available in 
individual PDFs on the PCSB website and DC PCSB produces a guide in multiple 
languages that explains the PMF and school results to families. Nevertheless, 
public-facing PMF reports could be substantially improved for a family audience 
that wants to know more than a tier rating: Reports are not written in parent- 


friendly language and may be difficult for families to interpret or compare. 
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MySchoolDC, D.C.’s unified enrollment system across public charter schools and 
district schools, links to the PMF results, but now emphasizes new ESSA-aligned 
STAR ratings created by the Office of the State Superintendent of Education. 
STAR ratings apply across charter and district-run schools. Currently, DC PCSB is 
considering how the PMF may evolve as the new STAR rating system across D.C.’s 


different kinds of public schools plays more of a public-facing role. 


New York City — School Quality Reports 


New York City’s school quality reports debuted under Mayor Michael Bloomberg 
as an accountability-focused school ratings system. More recently under Mayor 
Bill de Blasio, system leaders have significantly revised the reports to eliminate 
a summative rating, and integrate academic outcomes with observational and 
survey-based metrics. Schools are rated on seven dimensions of the city’s 
“Framework for Great Schools” — one rating is based on student academic 
achievement, the others draw from site reviews and school surveys. Metrics 
include test performance, progress, student postsecondary outcomes, and 


student experience (in the form of student surveys). 


System Management and Accountability 


The system was originally designed for this purpose under the leadership of 
Mayor Michael Bloomberg; it has moved away from this use case in its current 
iteration under the leadership of Mayor Bill de Blasio. It is unclear how these 
reports currently influence system management action citywide; practices may 


differ in different parts of the district. 


School Continuous Improvement 


This is the primary use case for this SPF. Detailed reports are aimed at school 
leaders, benchmarking performance against city averages and demographically 
similar schools. Additional resources for school leaders provide guidance on 
how to translate school quality report data into local improvement plans. Despite 
this emphasis, the sheer volume of metrics and the annual or longer turnaround 
on most measures may make it difficult for school leaders to prioritize areas for 


attention and change, and translate SPF data into timely action. 
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Family and Community Information 


School quality reports were not originally designed with families in mind, but New 
York City has made some recent changes to make the reports more family-friendly. 
Online school quality snapshots present ratings and data in a more simplified 
format, available in multiple languages. But the system is still complex, even ina 
simplified format, and may not clearly communicate relative school performance 
and strengths to families. Results are also not integrated with enrollment and 
school choice processes, which may mean relatively few families use these 


reports in practice. 


New Orleans — Louisiana School 
Performance Scores 


The state of Louisiana has rated schools since 1999, and introduced letter grades 
in 2011, with significant revisions in 2013 and 2018. Ratings are fairly simple. A 

to F grades are based on only two metrics at the elementary level: achievement 
on state tests as measured by an assessment index (75%), and state test growth 
(25%). The introduction of growth was one of the biggest changes in the 2013 
revision. Unlike some other systems, Louisiana includes scores on science and 
social studies assessments in additional to English and math. From 2005 to 2017, 
school performance scores played a key role in charter school expansion and 
closure decisions made while the city schools were mostly under the control of 
the state-run Recovery School District. Beginning in 2018, control of New Orleans’ 
schools transitioned back to the locally elected Orleans Parish School Board (OPSB). 
Now known as NOLA Public Schools, the school district continues to use the state 


SPF to make local school management and charter authorizing decisions. 


asa 
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System Management and Accountability 


Louisiana’s school performance scores play a role in system management at the state 
and local level, including NOLA Public Schools’ charter school authorizing and oversight 
process. At the state level, schools rated D or F for three consecutive years are 
identified for comprehensive school improvement under ESSA. At the local level, highly 
rated schools are eligible for longer charter renewal timelines, while persistently low- 
rated schools are ineligible for renewal and may be subject to early charter revocation. 
A high proportion of schools in New Orleans score at the lower end of the scale, with C 
or D grades. To differentiate among lower-rated schools and make renewal decisions, 
NOLA Public Schools created additional standards and policies on top of the state 


performance scores that emphasize school progress and growth over time. 


School Continuous Improvement 


School performance scores are not designed or intended to serve as a continuous 
improvement tool at a school level: They focus on high-level student outcomes. The 
SPF signals how a school is performing relative to outcomes goals, but it does not 
provide the type of data school leaders would use to assess specific causes of high- 
level outcomes or to make detailed instructional or operational decisions. However, this 
system is unique in that the state sees educators as a primary audience. State leaders 
want to clearly convey the importance of every student’s mastery of grade-level content 
aligned to college and career readiness via the school performance scores. This is the 


only SPF in our analysis that emphasized educator audiences in this way. 


Family and Community Information 


Clarity on school performance for families is another rationale for the simplicity of A to F 
grades and the choice to include a small number of metrics in the Louisiana SPF. When 
NOLA Public Schools, under the leadership of OPSB, considered how and if to use the 
state scores locally, it found that families and local stakeholders were familiar with the A-F 
grades, and valued stability in how schools were graded. School performance scores 

are part of school profiles on EnrollINOLA, the unified enrollment system for charter 
schools in New Orleans. The family- and community-facing school information website, 
LouisianaSchools.com, stands out for readability and clear design choices meant to 
address families’ priorities as well as ESSA requirements. The school district is in the 


process of designing local school profiles that include SPF scores along with other data. 
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Denver — School Performance Framework 


Denver’s SPF debuted in 2008 to measure school performance across the district, 
particularly as the district shifted to include more charter schools, innovation schools, 
and autonomous neighborhood schools. Denver uses a five-point color-coded 

rating system based on state test score growth and performance, student and family 
satisfaction surveys, attendance, and postsecondary readiness in high schools. A 
separate academic gaps score for historically underserved student groups interacts 
with the ratings — a school cannot earn the highest ratings if its academic gap 
performance is substandard. DPS has just begun a process to “Reimagine the SPF,” 
which will be led by a Quality Schools Committee composed of DPS stakeholders 


including school leaders, teachers, and family and community members.?° 


System Management and Accountability 


Denver’s SPF was primarily designed for system management, and differentiated 
services and interventions based on the SPF scores are baked into many aspects 

of central office operations. School ratings play a significant role in decisions about 
school closure, restart, levels of school autonomy, and charter school renewal. But 
policies, processes, and methodologies around the ratings are complex, and are not 
easily understood or fully transparent to the public. In some cases, changes in the 
system have made results fluctuate, damaging the public’s perception of the SPF 


and its use for consequential decisions about schools’ futures. 


School Continuous Improvement 


The SPF is intended in part to serve as an improvement tool that individual school 
leaders may use to inform and shape local school improvement plans. DPS sees 
the SPF ratings as a tool to drive school leaders’ attention to key metrics such 

as early literacy or academic gaps among student subgroups. According to DPS 
leaders, some critical feedback from school leaders over the years indicated that 
the SPF may be too complex, and the data in it not timely enough, to serve day-to- 


day continuous improvement purposes.?° 
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Family and Community Information 


The Denver SPF was always aimed to serve as a tool for public transparency about 
school performance, and it has made changes over the years to better serve this 
use case. Early on, parent advocacy groups prompted district officials to make 
changes to the system to make it easier for families to understand, including the 
introduction of the color-coding system to help parents distinguish between ratings. 
DPS has a unified enrollment system and produces an annual Enrollment Guide 

for families with ratings from the SPF. DPS includes these ratings to help parents 
differentiate among school options during the school selection process. Parents can 


also view this information online, using the district’s online school finder. 


As mentioned above, questions about the rating methodologies and changes 
to the SPF over the years may have damaged public perception of the SPF’s 


credibility, and made it less comparable over time. 
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Lessons From Existing SPFs 


he lessons that follow are aimed primarily at system leaders looking to 

design or adopt their own SPFs, or change an existing system, with an 

emphasis on cross-cutting themes and takeaways from the five SPFs in 
our analysis. These five SPFs each have strengths, weaknesses, moments of 
transition, and ongoing challenges to address moving forward. These real-world 
test cases provide insight that local leaders interested in developing or revising 


SPFs should consider as they seek answers to some key questions, such as. 


Why might a system need an SPF? 


To support school-level autonomy and accountability 


If a system is shifting its governance style to focus on school-level autonomy 

and accountability, an SPF primarily focused on system management and 
accountability could be useful. In Chicago, legislation that brought about mayoral 
control over schools also mandated the creation of a local SPF, which the city uses 
to determine a school’s level of autonomy under the oversight of a local school 
council. In D.C., as the charter sector grew, the DC PCSB recognized the need for 


a common tool across schools to guide authorizer decisions. 


To support data-informed school improvement 


If a system has a goal of improving all schools through data-informed action, an 
SPF focused on school continuous improvement may be a useful tool for school 
leaders. Only one of the SPFs in our analysis, New York City, was primarily focused 
on school continuous improvement — although when the SPF began, it was more 
focused on accountability. Today, New York City’s SPF aims for a comprehensive 
framework that includes extensive academic and environmental metrics, and 

is flexible enough to be relevant and functional across 1,800 different schools. 
Refining the system to serve this purpose, and implementing it with fidelity, is an 
ongoing process. New York has created alternative views of its SPF for family, 
school leader, and system leader audiences, with different levels of detail and 
areas of emphasis. 


In these examples, there is a key tension for SPFs that aim to support school 
continuous improvement: School leaders need much more granular, frequent 

data to support ongoing decisions than families or system leaders tend to prefer. 
This use case may be less compatible with the other two. But a community might 
consider parallel or adjacent data tools linked to a public-facing SPF that provide 
school leaders with the information they want. For example, Chicago’s SPF is not 
primarily designed for day-to-day school action, but school leaders get their annual 
SPF results broken down into a student-level roster; this enables them to look at 
the data in a different way. 


To support families’ choices and empowerment 


Family- and community-focused SPFs tend to emerge where there is a relatively 
high degree of choice and parents need a clear and common basis on which to 
compare schools. For instance, Louisiana’s new family-facing school finder was 
noted as an exemplar of usability and clarity in Data Quality Campaign’s national 
scan of state school report card websites.?” Additionally, the local school board 
plans to debut local report cards that include state data alongside other school 
information. This is extremely important in a choice-driven school system like 
New Orleans. 
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Local SPFs should aim 
to clarify, not confuse 
the understanding of 
school performance 
among stakeholders, 
including the school 
community and family 
and community 


members. 
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Beyond informing the process of choosing a school, family- and community- 

facing information can be a valuable asset in school improvement. If families and 
community members are better informed about school performance and quality, 
they can also be more empowered to engage with school improvement processes 


and advocate for community priorities with school and system leaders. 


What if a state already has a school rating 
system — is that enough? 


Regardless of the use case(s) that may prompt local leaders to consider developing 
an SPF, any SPF should be additive, not redundant or in conflict with state 
accountability and rating systems where they exist. Locally developed SPFs have 
the opportunity to provide a more nuanced picture of school performance that 
reflects local goals and community priorities. Local leaders also have greater on- 
the-ground responsibility and decision-making power than state leaders, which 
changes the scope and purpose of a local SPF. Local SPFs should aim to clarify, not 
confuse the understanding of school performance among stakeholders, including 


the school community and family and community members. 


Many of the SPFs profiled in this work originated during No Child Left Behind, as 

a response to federally mandated school ratings leaders felt were too blunt, and 
not relevant to local goals and context. These same systems are now grappling 
with whether and how their longstanding local systems should adapt to new 
state-designed ratings and reports under ESSA. ESSA state ratings tend to 

include a wider variety of data elements, such as academic growth on state tests, 
attendance, or school environment surveys. With this wider array of measures in 
state systems, local leaders must consider what purposes additional information 
will serve. Even so, all of the local system leaders we interviewed felt that their local 


SPF remained relevant even with a new state system. 


Developing and implementing a high-quality SPF requires significant investment 

of time and resources, and local leaders should be certain that a new SPF is 
necessary. In a state with an existing rating or report card system, adapting the 
state accountability ratings might serve certain use cases well and could streamline 


the process of SPF creation. For example, in New Orleans NOLA Public Schools 


uses state performance scores as a local school performance framework to 

guide decisions including whether a school is eligible for charter renewal. Local 
leaders considered creating their own SPF several years ago, but in the course of 
community outreach, they found that many parents and stakeholders were familiar 
with the state system and felt it was credible. Although there are some drawbacks, 
NOLA Public Schools has been able to adapt the state SPF to local needs. 


In determining whether and how to design and implement a local SPF, local leaders 


should evaluate existing state rating and reporting systems and consider: 
e The extent to which state systems reflect local goals and priorities 


e The level of adaptation necessary to serve the goals local leaders would have for 
an SPF 


Should an SPE aim to serve all three use 
cases? How can an SPF serve more than 
one user group effectively? 


An SPF does not necessarily need to serve all three use cases, and it is unlikely 
that any SPF would be able to completely meet the needs of system leaders, 
school leaders, and family/community members at the same time. Several system 


leaders described trying to “do it all” as deeply problematic for system design. 


Trying to serve all stakeholders and use cases equally, without clear priorities, can 
lead to a muddled system. Rather than aiming to do everything at once, at the 


outset of an SPF process, leaders interested in an SPF should: 
e Clearly understand their goals and the unmet needs the SPF will serve 


e Map out and assess other systems currently serving the needs of each user 
group — for instance, existing data systems for school leaders, or resources 


for families choosing a school 


e Differentiate between primary and secondary use cases the SPF will serve — 


or decide which use cases the SPF will not serve 
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priority use Cases. 
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Design choices should reflect the priority use cases. Where tradeoffs must be 
made, the defined priorities should drive decisions. For example, an SPF designed 
for systems management may require a detailed array of metrics on student and 
subgroup performance to help leaders assess schools against system goals. 
However, that may not be reflective of the kind of information families need or 
want. Where use cases are in conflict, leaders should prioritize their top use cases 
in design decisions. This is not to say that systems cannot serve more than one 


purpose, but conflicts will arise, and leaders may need to compromise. 


Some combinations of SPF use cases may fit better together than others. Several 
of the systems we examined have system management uses as their primary 
purpose, and sharing information with families as a secondary use. Both system 
leaders and families share an interest in student progress over time and high- 
level student outcomes. In these cases, there is often a parent-facing view of 
information that displays ratings first, offers a deeper level of detail if desired, 
and may include information that is not part of the SPF — like programmatic 
offerings or information on the school environment. In these cases, customized 
communications strategies can leverage the same or similar underlying data, but 


differentiate how those data are presented. 


The most difficult use case to integrate with others is school continuous 
improvement. The level of detail and types of diagnostic and outcome data a 
school leader needs to create an effective action plan and drive decision-making 
is much different from the higher-level outcomes-focused metrics that are most 
actionable for system leaders and parents. The only SPF we examined that placed 
school continuous improvement as a primary use case was New York City, and 

it is unclear how usable that system is for school leaders in practice. New York 
City’s SPF does not have an overall rating, and it measures schools across seven 
dimensions, only one of which is academics. The breadth and depth of data 
included even in the simplest version of the New York City school quality reports 
is unlikely to be easy for parents to digest, or point to clear action steps for system 


leaders. 


The “owner” of an SPF can shape design and uses. For example, the DC Public 
Charter School Board and the Orleans Parish School Board are clear that as 
authorizers they do not carry out improvement actions at the school level, and 
thus their SPFs are designed to focus on student outcomes and growth, not 


detailed diagnostic measures school leaders might use to develop, implement, 


and track improvement plans. Chicago, Denver, and New York City’s SPFs include 
many kinds of schools, so these SPFs have different oversight and management 


implications for different schools. 


Use case priorities can also shift over time. Leaders in Chicago recently adapted 
their system into a more family-friendly online format, after realizing that families 
increasingly wanted to use and understand their system-focused SPF. D.C. is 
considering whether the charter school PMF should still attempt to be family- 
facing if the state system, called the STAR rating, is more closely aligned to that 


use case. 


One critical thing for system leaders to bear in mind is that user needs will not 
disappear just because the SPF is not designed to serve them. And as a practical 
matter, anything a school system does to measure and assess school performance 


should be transparent and accessible to the public. 


If leaders want to design an SPF that serves more than one user group, there are 


a few key considerations: 


1. Engage all stakeholders proactively and authentically. Engagement should 
align with users’ different roles and priorities in the system. This step is key to 
understanding different needs and identifying potential tensions or conflicts 
early. Stakeholder engagement should not come late in the process, after key 
decisions have already been made, but should provide authentic opportunity 


for actionable input.?® 


2. Anchor the system and metrics in common, coherent goals for students. 
Different user groups may be more or less interested in certain kinds of 
metrics and benchmarks. An SPF designed to serve multiple users may need 
to compromise, or allow for displays of data that are customized for different 
users, but there should be coherence across the use cases such that parents, 
school leaders, and system leaders receive the same messages about 


schools’ relative and absolute levels of performance. 


3. Translate SPF results for different audiences. A single PDF report will not 
serve multiple user groups well. The most successfully communicated multiuse 
SPFs design different views of data specialized for families, schools, and 
system leaders. Systems can be even more transparent by releasing easily 
analyzable data to external researchers and organizations, which could bring 


about new insights. 
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4. Support action. SPFs are not solely informational — they should be used to 
drive action among users. If an SPF is meant to support parent choice, families 
will need more resources, such as a well-functioning enrollment system and 
one-on-one support and counseling in order to understand their choices. If 
an SPF is meant for school leaders to use in improvement, those leaders will 
need coaching and professional development to use information effectively in 


their day-to-day and long-term strategies. 


What are the pros and cons of a 
summative rating in an SPF? 


The considerations for whether or not to include a summative rating in an SPF are 
different from that of a state-level accountability system, because local SPFs tend 


to have different goals and theories of change than a state system. 


The question of whether an SPF should have a summative rating provoked 
strong reactions among system leaders and advocates we spoke with on this 
topic. Proponents, including leaders of the four systems we profile that include 

a summative rating, feel that a clear rating is essential for a transparent family- 
facing SPF, and for clear links to system leader decision-making — differentiating 
between high- and low-performing schools across a variety of metrics. Advocates 
for parents usually agreed that parents find a summative rating helpful in using an 


SPF, especially for school choice decisions. 


On the other hand, system leaders who are not in favor of a summative school 
rating describe summative ratings as often misleading, distracting families from 
digging deeper into their own priorities for different school quality indicators, and 
encouraging school leaders to take an overly narrow and prescriptive view of 
improvement, especially around test scores. And if state accountability systems 
already include a rating, it will be difficult for a new local system to enter the public 


conversation without causing confusion. 


Differing views on the pros and cons of summative ratings come down to SPF 
values, goals, and implementation decisions, so there is not one absolutely right 
answer. An SPF with a summative rating is not automatically clear — for example, 


Chicago’s categories of 1+, 1, 2+, 2, and 3 do not communicate much at a glance. 


And confusion about the methods and business rules that go into a summative 
rating can obscure meaning and reduce credibility, as is currently the case in 
Denver. An SPF without a summative rating will need to work harder to guide 
family audiences through the data, and ensure that the policies and processes of 


considering SPF performance in system decisions are clear. 


What does it mean to align metrics and 
methods to system goals? 


Across the three use cases, aligning metrics and methods to system goals is 

critical. A logical and effective SPF should clearly link up with long-term goals 
for students across schools. The five systems we examined provided various 
examples of that principle in practice. 


In Chicago, system leaders wanted to emphasize academic growth. So the city 
adopted an assessment in grades 3-8, the NWEA MAP, which measures growth 
and performance across grade levels, and weighted a substantial proportion of 
their elementary/middle school ratings on growth percentiles and performance 
among priority groups. They also wanted to communicate the importance 
of school culture and climate, so a school climate survey makes up a small 


percentage of school ratings across the system. 


Louisiana’s system heavily weights achievement on state assessments in large 
part because state leaders want educators to focus on helping every student 
master grade-level content. Using an assessment index that credits improvement 
at each level of achievement instead of a proficiency percentage also helps 
encourage growth across the full performance spectrum — with a “percent 
proficient” measure, educators may have incentives to focus most on students 
closest to the proficiency bar. The choice to weight assessment performance at 
75% of schools’ scores comes with potential tradeoffs: A rating based heavily on 
assessment performance instead of growth is likely to penalize schools whose 
students come in below grade level, including schools serving high proportions 
of at-risk students. But it also serves to unify system and school leaders around a 


clear goal. 


School Performance Frameworks: Lessons, Cases, and Purposeful Design [37] 


Including a large 
number of metrics can 
dilute the importance 
of measures most 
closely aligned to 
top-level goals in any 


summative rating. 


[38] Bellwether Education Partners 


What is the right number of metrics or 
indicators in an SPF? 


There is no perfect number of metrics or indicators in an SPF — like almost every 
other design consideration, the right choice depends upon the definition of 


success and use cases in mind. But there are some tradeoffs to consider. 


It may be tempting to include every important data point available in the SPF. 
However, too many metrics can create confusion around what is most important, 
especially for family and community user groups. Including a large number of metrics 
can dilute the importance of measures most closely aligned to top-level goals in any 
summative rating. Weighting methodologies can compensate for this to a certain 
extent, but even then including a large number of metrics that contributes very little to 


a summative rating raises questions about what matters most and where to focus. 


In addition, as leaders consider metrics to include, ensuring that metrics contribute 
unique information to the system is important. If two or more metrics are closely 
correlated with one another, they can interact and skew SPF results when 
combined. When considering a long list of potential metrics, assessing the extent 
to which any one metric adds new information to the overall system can be one 


criterion for narrowing the list. 


On the other hand, too few metrics can paint an imbalanced and limited picture 
of what school success looks like, especially if SPFs focus narrowly on just a few 
ways of measuring academic success. This was the case with NCLB ratings, which 


spurred the creation of several long-standing local SPFs. 


System leaders should look for compromise solutions that bring nuance to an SPF 
without sacrificing clarity. For example, subgroup performance and/or achievement 
gap measures in an SPF bakes equity into the definition of a “high-performing” 
school. But including subgroups in every metric of an SPF would automatically 
multiply the total number of metrics eight-fold or more. SPFs we examined took 
some creative approaches to this topic: Chicago places additional weight on 
academic growth and performance for a few priority student groups, such as English 
learners and special education students. In Denver, in addition to the overall score, 
each school receives a separate academic gaps score intended to measure how 
well schools are driving growth among historically underserved student groups. 


Schools with large achievement gaps are ineligible for the top school rating. 


One of the most 
important qualities to 
ensure a sustainable 
and durable SPF is 
stability. 


How do I know my system is ready for 
an SPF? How long will the process of 
creating one take? 


Certain inputs and resources are critical for an SPF, such as reliable data sources, 
a clear vision for system goals, and buy-in (or potential for buy-in) across key 
stakeholders. As discussed above, the needs and intended uses for the SPF 


should be clear at the outset. 


Leaders should also consider the practical side of SPF creation — do you have the 
staff capacity and expertise to lead this process? Once the SPF is created, do you 
have the resources to create communications tools or reports and support users 


to understand the system and its goals? 


One mistake several system leaders of current SPFs highlighted was getting 
too caught up in the metrics stage of the process. A leader could spend years 
debating the pros and cons of different growth metrics or ways to measure 


achievement gaps, especially if the goals and vision for the SPF are unclear. 


Creating or substantially revising an SPF will take at least one school year once 
all the prerequisite pieces are in place, and may take longer than that. System 
leaders should set realistic benchmarks and deadlines and agree on key decision 


roles in advance in order to keep the process moving forward. 


What can I do to ensure my SPF is 
sustainable and durable over time? 


One of the most important qualities to ensure a sustainable and durable SPF 

is stability. The key metrics in an SPF should stay relatively constant over time. 
New metrics or methods should only be introduced after extensive consideration 
and engagement. Results should not fluctuate wildly without reason, and the 
actions taken based on the SPF should be clear and consistent. Chicago leaders 
credit the stability of their SPF since 2013 with creating strong credibility among 
different stakeholders and user groups, especially school principals. In contrast, 


Denver’s changes in methods and additional metrics have had some unforeseen 
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impacts on the distribution of school performance across the ratings, which 


could reduce the credibility of the ratings. 


Public support for an SPF can help improve durability. If parents and community 
members value the SPF and see it as a credible and useful tool, they won’t want to 
see it eliminated. Offering community members ways to provide input early in the 
SPF process can foster a sense of ownership and show that the SPF is responsive 


to community priorities. 


There is a balance between stability and responsible cycles of improvement and 
evaluation. There should be consistent cycles of analysis and evaluation after 

the SPF goes into effect to make sure that it is functioning as intended, and to 
identify areas for improvement or additional user supports. Changing an SPF 
already in place should be a deliberate process — especially if the SPF results are 
tied to decisions like school closure. In Louisiana, after the state adopted higher 
standards and assessments resulting in lower levels of mastery across the state, 
it continued to rate schools, but the state limited the distribution of ratings for 
several years.”? This meant that schools and state leaders had time to see how 


scores would shift under a new system, while maintaining an incentive to improve. 


SPFs, like many other high-profile initiatives in education, are subject to changes 
in leadership. For example, New York City’s system underwent a wholesale 
overhaul after a mayoral and chancellor transition. Even SPFs mandated by state 
law, like Chicago’s and Louisiana’s, could be substantially changed in the future 
by new leadership. In New Orleans, when school oversight transitioned from state 
control under the Recovery School District back to the local school board (OPSB), 
the school board kept the state SPF as part of its system, but took a fresh look 

at how growth and progress factored into authorizer decisions. A leader looking 
to create an SPF that withstands leadership changes should focus particularly 

on stakeholder engagement and buy-in. If multiple stakeholders in a system see 
the value, validity, and relevance of an SPF, it will be significantly less likely to fall 


victim to politics or backlash. 


SPF Planning Guide 


The lessons learned from the research, analysis, and discussion above, as 

well as the examples provided by five longstanding, local SPFs, point to 

some key action recommendations for local leaders considering an SPF. 
These recommendations span the full lifecycle of an SPF, from the initial decision 


to create this tool, to the ongoing work of implementing, communicating, and 


refining the tool over time. 


STAGE 3 


STAGE 1 STAGE 2 


Understand Design and Implementation, 
Evaluation, 


Decision-Making 


the Landscape, 
Set Goals and 
Priorities 


and Ongoing 
Sustainability 
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Understand the Landscape, 


STAGE 1 ‘ ‘io 
—— Set Goals and Priorities 


Engage stakeholders to understand 

1 what information they need about school 
performance, and what information they 
already have. 


3 Gather inputs and resources. 


e Assess the time, information, authority, and 
capacity within the system to create an 
e Assess data systems and resources focused SPF — identify where additional capacity or 
on school performance available to different resources may be needed. 


users at a state or local level. e Map out available data and places where 


e Evaluate whether those tools are meeting new or improved data may be needed. 
local needs. 


0 Is this information coherent and sufficient — 
to drive action? 4 Make a decision as to whether to pursue 


a new or substantially revised SPF. 
oO Is information available transparently, 
accessibly, and equitably to those who 


it? 
mostneed it: e Facilitate stakeholder alignment on the need 


for an SPF, the primary goals and audiences, 
and the intended uses. 


Define clear goals and priorities 


Communicate that decision, the rationale 
for an SPF. ’ 


behind it, and the process for design to user 
groups. 


e Discuss what uses and users an SPF would 
serve. System leaders? School leaders? 
Families and community members? Other 
stakeholders? 


o Align use cases to system strategic goals. 


o Decide which users and uses will be 
primary, and which will be secondary, 
based on system goals and user needs 
established during engagement. 


e Set goals, outputs, and outcomes against 
which the SPF’s success will be evaluated. 
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STAGE 2 Design and Decision-Making 


Build the SPF, learning from lessons, 3 Create differentiated reports, tools, and 
examples, and information about SPFs resources to serve intended audiences. 
with similar goals and use cases. 


e Choose valid and reliable metrics that align 
with system priorities and user needs. 


e Design benchmarks, weights, or tiers to 4 Engage and take feedback from intended 
differentiate school performance, with user groups at key decision points. 
primary and secondary user groups in mind. 


e Decide whether and how to assign a 
summative score to schools. e Differentiate engagement strategies based 
on users’ key concerns and needs. 


e Consider equity and inclusivity — the loudest 
voice in an argument may not be the most 


Model potential choices, and : = : 
2? representative or critically important. 


consider tradeoffs. 


¢ Communicate process and decision points 
along the way to the public and to each 


e Use real-world system data to test whether user group, considering their needs and 
and how the SPF works as intended, engagement with the system. 
and anticipate potential problems and 
weaknesses. 


e Where conflicts or questions arise, make 
decisions based on previously set goals and 
priorities. 
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STAGE 3 


1 Pilot the completed SPFs with users, and 
set a timeline for full implementation. 


e Weight the value of advance knowledge and 
predictability for school leaders, families, 
and other community members against 
potential advantages of having the SPF up 
and running more quickly. 


Evaluate the completed SPF 
against intended impacts, with 
feedback from users. 


e Integrate an evaluation plan, key goals, and 
metrics into the launch process. 


e Align evaluation goals and indicators with 
intended use cases. 


e Monitor for unintended consequences or 
unanticipated uses of the SPF, including 
equity implications of SPF results and uses 
(e.g. disproportionate effects on certain 
subgroups of students). 
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4 


Implementation, Evaluation, 
and Ongoing Sustainability 


Establish review processes and cycles 
to reassess the system at regular 
intervals and enable responsiveness to 
shifts in users’ needs, local conditions, 
and system priorities. 


Ensure the SPF remains useful and relevant 
and that policies tied to the SPF make sense. 


Avoid sudden, unintended shifts in SPF 
results, and carefully model any proposed 
changes or additions to the system. 


Communicate needed changes clearly to 
stakeholders, including the rationale for and 
impact of changes. 


Integrate the SPF into action, with clear 
and documented processes and policies. 


Document action steps that can or will be 
taken based on SPF results, including clear 
policies for how SPF results will be factored 
into decisions. 


Support user groups with guidance, 
outreach, and communications tools to 
promote understanding and transparency 
regarding how SPF results will be used. 


Detailed SPF Profiles 


Schools and Enrollment 


SPF Origins 


Years Operational 


Purpose 


Communication 
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Chicago Public Schools’ (CPS) School Quality Rating Policy (SQRP) applies to 644 
schools enrolling over 360,000 students.°° Of those, 517 schools are district-run, 


121 are public charter schools authorized by CPS, nine are contract schools. 


CPS is a mayorally controlled district, meaning the mayor appoints members of the 
school board, who approve the CEO of the school district. The same state law that 
established mayoral control in 1995 also requires CPS to monitor the performance 


of all schools.*! The SQRP fulfills this requirement. 


The current iteration of the SQRP has been in use by the Department of School 
Quality Measurement and Research since the 2013-14 school year. Prior 
iterations of CPS’ performance management system did not apply to charter 


schools. 


CPS uses the SQRP to measure annual school performance across the district, 
magnet, and public charter schools it oversees and authorizes. SQRP results guide 
decision-making processes around school actions, system strategy, and turnarounds; 
determine the level of interventions and school oversight, particularly the level of 


control given to local school councils; and affect charter renewal decisions. 


CPS communicates the results of the SQRP in four ways: direct principal notification, 
direct local school council notification, the CPS website, and online school progress 
reports. In addition to SQRP reports, principals receive a roster report with the 


performance of each individual student on applicable SQRP metrics. 


Metrics, Weights, | CPS uses different metrics depending on schools’ grade levels. Sixty percent of 
and Ratings elementary and middle schools’ ratings derive from student performance and 
growth on NWEA MAP reading and math assessments (MAP is an adaptive, interim 


assessment that CPS administers three times a year in addition to Illinois state 


tests), with an emphasis on growth and closing gaps for low-performing student 
groups. The remaining 40% is based on student attendance, qualitative survey 


data, growth in English language proficiency among English learners, and a data 


quality measure. High school scores come from a wider variety of data sources, 


with an emphasis on college readiness metrics such as the SAT, graduation, 


college/career credentials, and college enrollment rates. 


SQRP Metrics and Weights 


Elementary/Middle Schools High Schools 
Metric Weight Metric Weight 
Student growth on NWEA MAP 25% Student growth on PSAT/SAT 20% 
Student attendance 20% Growth of priority groups on SAT 10% 
Growth of priority groups on NWEA MAP 10% Student attainment on SAT 10% 
oe fatlenal average 10% Student attendance 10% 
5Essentials Survey 10% Freshman On-Track rate 10% 
ae OnLNwen a 10% 4-year cohort graduation rate 10% 
ee ell on en? 5% Early college/career credentials 5% 
ELL language development growth 5% 1-year dropout rate 5% 
Data quality 5% College enrollment 5% 
College persistence 5% 
5Essentials Survey 5% 
Data quality 5% 
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For each of the metrics a school can earn between one and five points. Metric 
scores are then weighted and averaged, with test score growth receiving the 
heaviest weight and therefore greatest influence on the summative score. The 
weighted average, which also falls between one and five points, is then used to 
determine a school’s school quality rating and accountability status as explained in 
the chart below.* There are some cases where a school may qualify for a higher 
rating than the full array of metrics would yield if the school’s MAP or PSAT/SAT 
scores are particularly high.2° The CPS CEO also has the authority to grant a one- 
time, one-year extension to a level 1 or 1+ school that experiences a leadership 


change or other significant event that affects school performance.*° 


Overall Score School Quality Rating Accountability Status 
4.0 or more Level 1+ Good Standing 
Between 3.5 and 3.9 Level 1 Good Standing 
Between 3.0 and 3.4 Level 2+ Good Standing 
Between 2.0 and 2.9 Level 2 Provisional Support 
Less than 2.0 Level 3 Intensive Support 


System Management and Accountability 


The SQRP was primarily designed to serve system management purposes. 
Higher-rated district-run schools get more building-level autonomy, while lower- 
rated schools get more monitoring and intervention, and may be at risk of closure. 
Charter schools have a slightly different relation to the ratings, but performance on 
the SQRP is a clear factor in charter renewal and expansion decisions. Jeff Broom, 
CPS’ director of school quality measurement and research, described the SQRP as 
a system of reciprocal accountability that measures the progress of schools and 
the system as a whole. CPS has published a detailed SQRP handbook and a rating 
calculator tool, which make clear the purposes of the system and the relationship 
between ratings and school governance actions, metrics, and methods. The SQRP 


has been relatively stable since the city overhauled its SPF approach in 2013-14, 


which CPS officials believe has resulted in a high degree of credibility and trust 
from school leaders and community members around the ratings and the ways the 


central office uses them.?” 


Metrics in the SQRP align with system priorities and long-term goals, reflected 

in the district’s five-year strategic plan.*® The SQRP heavily rewards growth in 
elementary and middle schools. The assessment CPS uses, MAP, can measure 
student growth across grade levels, which can be especially useful when students 
are performing far above or below grade level. CPS’ decision to use MAP also 
insulated the rating system from instability as the state made changes in its 
assessment.*? In high schools, the SQRP clearly emphasizes postsecondary 
readiness, with six metrics explicitly related to college readiness, enrollment, and 
persistence, in addition to more common high school metrics like graduation and 
ninth-grade credit accumulation. 


CPS officials describe a research-driven process for assessing and adopting 

valid and reliable metrics for the SQRP, guided by an external advisory group of 
researchers and experts. For example, the 5Essentials survey, a measure of school 
environment and culture, and the Freshman On-Track measure, a measure of 
freshman course performance, were both developed at the University of Chicago 
Consortium on School Research. 


In terms of equity in the ratings, CPS officials say the system has come a long 

way from pre-2013 SPF iterations that closely correlated with racial and economic 
demographics in schools. An emphasis on growth metrics in elementary and 
middle schools allows schools serving students behind grade level to demonstrate 
progress. High school metrics are less reliant on growth, and therefore, more 
closely correlated with incoming student status and demographics. Officials say 
that they still have further to go to ensure that ratings fully reflect performance 
differences among schools, rather than demographic differences. CPS recently 
hired a chief equity officer, who will be working with the SQRP team to audit the 
system from an equity perspective. 
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School Continuous Improvement 


The SQRP is not designed or intended to serve as a continuous improvement 

tool for schools: It does not differentiate between diagnostic metrics and key 
outcomes, and many metrics, especially at the high school level, are only available 
after the end of the year. A few are on multiyear time lag (e¢.g., college attendance 
and persistence). But CPS has taken steps in recent years to help school leaders 
better use their SQRP results to inform school-level action, and prioritize areas for 
improvement. For example, CPS gives principals a roster report of every student’s 
performance on the measures that apply to the SQRP, such as assessment 

growth and attendance. This way, school leaders get a much more fine-grained 
perspective on the trends among their students. 


Another way in which the SQRP communicates priorities to school leaders is in 

its metrics. Both high school and elementary school ratings include a category 

for “growth of priority groups,” which CPS defines as English learners, students 
with an individualized education program (IEP), and African-American and Latino 
students. By placing extra weight on growth among the lowest-performing student 
groups, CPS rewards schools doing the most to close achievement gaps for 
traditionally underserved students. 


But in a broader context, school leaders are on the receiving end of mixed 
messages from the state and district leaders about what student outcomes 
matter most, and where they should spend their time. The state of Illinois recently 
introduced a new accountability system under ESSA using different metrics. 

CPS’ system does not use the state test as a factor in elementary SQRP scores, 
but Illinois’ accountability system for elementary schools is heavily based on 

state test results. This year when the state system rolled out, some of CPS’ top- 
rated schools were marked among the states’ lowest performers.*° CPS officials 
served as advisers to the state as they created the new system, but there are no 
current plans to align the SQRP with the state or adopt the state test as an SQRP 


measure.” 


Family and Community Information 


The SQRP was not originally designed with families in mind, but as CPS has 
introduced more public charter school and intra-district school choice options 

for families, the district has increasingly recognized the need for transparent, 
shared information across schools. CPS’ online school profiles stand out for 
readability and clear design choices meant to address families’ priorities.4* The 
overall school rating is at the top of the profile. Information about the size of the 
school, leadership, programmatic offerings, and transportation is also accessible. If 
families choose to dive deeper, growth is at the top of the school progress report, 


in line with the weight it has in the system. 


Eight language translations are integrated into CPS’ school progress reports, so 
that parents reading in a language other than English see the same information. 
CPS also offers information about the SQRP in parent-friendly formats, such 

as an explainer video in English and Spanish. Many key terms for metrics have 
definitions and links to more resources available with a click in the school 


progress report. 


There are some places where family-friendly communication falls short. CPS 
schools are graded on a five-point scale, but the labels for that scale are not 1-5. 
Instead, they are 1+, 1, 2+, 2, 3. In other places on the school progress report, 

a five-point scale has different meanings, such as Far Above Average, Above 
Average, Average, Below Average, or Far Below Average in the category of 
student growth.* The proliferation of scales and rating meanings within the SQRP 


could make the system confusing for a non-specialized audience. 
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SPF Origins 


Years Operational 
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Performance Management Framework 


The DC Public Charter School Board (DC PCSB) is the public charter school 
authorizer for the District of Columbia. The members of the board are nominated 
by D.C.’s mayor and approved by the D.C. City Council. DC PCSB authorizes 123 
public charter schools enrolling 43,000 students, just under half of the district’s 
total public school enrollment. D.C.’s charter sector is somewhat unique in that 

it includes 3- and 4-year-old preschool students as well as adult students. The 
Performance Management Framework (PMF) applies to all D.C. public charter 


schools. 


Prior to the PMF, DC PCSB evaluated each charter school on the different, 
individual goals captured in its charter. The board sought a more transparent, 
consistent way to measure and manage performance across the sector, identify 
high- and low-performing schools, and encourage all schools to improve student 


outcomes. 


DC PCSB first piloted the PMF internally in the 2008-09 and 2009-10 school years; 
in 2010-11 DC PCSB worked to revise components of the system and released 
results in fall 2011 with data collected in the previous year.“ Adult, alternative, and 


early childhood frameworks came later. 


Purpose 


Communication 


Metrics, Weights, 
and Ratings 


DC PCSB uses the PMF as a tool to set consistent performance standards, and 
communicate school performance to schools and families. The board uses the 
PMF results as a factor in charter expansion, renewal, or closure decisions.*° 
Families can use individual school reports to find out how their school is 


performing or guide school selection.*® 


PMF results are available to the public as static school quality reports on the DC 


PCSB website and in small on-the-go guides.”” 


The PMF assigns schools a rating based on a three-tiered system. The PMF 

uses different metrics depending upon the grade levels a school serves. Metrics 
emphasize student growth and achievement on state tests, and key “gateway” 
measures that correlate to longer-term outcomes for students, such as third- 
grade reading and ninth-grade credit accumulation. Public charter schools in D.C. 
have a variety of grade configurations, and accordingly, there are eight grade 
configurations under the PreK-8 PMF. There are also specialized PMFs for schools 
serving early learners (without students in grades 3 and 4), adult learners, and 


alternative schools. 


Each metric is worth a different number of potential points on a 100-point scale. 
Each metric has a floor and a target. The floor is the minimum value for a metric 
that will earn any points for a school. The target is the maximum value for the 
metric, which will earn the school full points. Schools between the floor and the 
target will earn incrementally more points. Performance above the target does 
not earn any additional points. Floors and targets are adjusted biannually for 
some metrics, while others are static. Schools are then identified as Tier 1 (65-100 
points), Tier 2 (35-64 points), or Tier 3 (0-34 points).*° 
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Elementary/Middle Schools (Without PreK Grades) High Schools 
Metric Weight Metric Weight 
Student progress 40% Student progress 15% 
(median growth percentile — (median growth percentile — 
ELA and math) ELA and Math) 
Student achievement 30% Student achievement 20% 
(proficiency) (proficiency) 
Gateway 10% Gateway 42.5% 
(proficiency on 3rd-grade reading and (9th-grade on track, 4- or 5-year graduation 
8th-grade math) rate, PSAT/SAT proficiency, college 

acceptance rate, AP/IB passage rate, rate of 

students earning college credit) 
School environment 20% School environment 22.5% 
(attendance and reenrollment) (attendance and reenrollment) 


System Management and Accountability 


The PMF was created primarily as a consistent, transparent tool for authorizer 


decision-making. PMF Tiers may encourage expansion, or trigger a high-stakes 


review process that leads to closure ahead of regular charter renewal timelines. 


Tier 1 schools are encouraged to expand and replicate. Tier 2 schools may be 


subject to closer monitoring and review.*? Schools with Tier 3 ratings may be 


subject to accelerated charter revocation (closure) based on the results of a more 
in-depth, high-stakes review. Schools scoring in Tier 3 for multiple years in a row, 
schools earning 0-20 points on the PMF scale, or schools in Tier 3 that appear to 
be declining in performance from one year to the next are at higher risk of charter 


revocation. 
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PMF tiers are not the only factor that could put a charter school at risk of closure, 
which complicates the relationship between the PMF and consequences for 
schools. Schools are held to the terms and goals of their individual charters, on 

a five-year review cycle. At the encouragement of the PCSB, many schools have 
adopted the PMF as a central part of their charter goals, while other schools 
have unique, individual charter goals that do not link to the PMF. Schools have 
the option to adopt the PMF as part of their charter goals and maintain additional 


goals that reflect their unique mission. 


One part of the high-stakes review process is a Qualitative Site Review (QSR). 

The QSR includes meetings with school leaders, unannounced school visits 
focused on classroom observation, and reviews of school board meetings and 
parent interactions.°° A QSR is part of periodic charter renewal procedures, but if 
a school earns Tier 3 status on the PMF, a QSR will take place the following school 
year. QSR results, PMF data, and other qualitative and quantitative data on school 


performance and operations are all part of a charter review and renewal process. 


The PMF emphasizes academic outcomes and growth, which make up 
approximately 80% of the elementary/middle and high school PMFs. Annual 
growth and outcomes in key “gateway” grades and subjects such as third-grade 
reading are more heavily weighted. Recent analysis from DC PCSB has found that 
these measures are highly correlated with school demographics and thus are not 
the best measures of school quality.*' As a result, DC PCSB is planning to either 


phase out these “gateway” measures, or determine a way to improve them. 


Because many students and families choose to enroll in a charter school rather 
than a neighborhood school or another option, the PMF uses reenrollment as a 
proxy measure for student and family satisfaction. Attendance is another measure 
of student and family engagement, and students’ opportunity to learn. In line with 
its goals as an authorizer (instead of a school district central office), DC PCSB 
intentionally does not include measures of school culture or environment beyond 
reenrollment rates and attendance, because it sees its role and authority as 


focused on outcomes. 
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The PMF model is more complicated for specialized schools. The PMF goes 
further than some other SPFs to recognize programmatic differences among 
schools by creating specialized performance frameworks for schools serving 
younger students, adults, and/or other specialized populations. Since the debut 
of early childhood, adult, and alternative PMFs, DC PCSB has debated ways to 
assess and tier these schools in comparable ways to schools serving students in 
grades 3-12, especially because outcomes metrics for younger and older students 
are often less standardized, and DC PCSB does not have the authority to mandate 
additional assessments beyond those required by law without schools’ consent 
(currently, the District of Columbia requires tests in grades 3-8 and once in high 


school). 


Some advocates and some charter school leaders say the PMF results correlate 
too closely with school demographics, and disadvantage schools serving high 
numbers of at-risk students.°? None of the metrics in the PMF is disaggregated 

by student subgroup. The PMF includes and places substantial weight on median 
growth percentile, which is less closely correlated to race, ethnicity, and income 
than overall achievement measures like proficiency. But it includes proficiency and 
other metrics that may tip the balance away from measuring schools’ impact on 
their students. To correct this issue, the PMF could do more to control for student 
population and adapt to schools’ differing models and missions, but this could also 
negatively impact the consistency and comparability of the PMF ratings for the 


authorizer’s purposes. 


School Continuous Improvement 


As mentioned earlier, DC PCSB does not see the PMF as a tool for school 
continuous improvement, but it does encourage schools to use it as a part of 
their official charter goals around student outcomes. Schools can choose to align 
their goals, action plans, and management decisions to the PMF, or hold it at 
arm’s length. To the extent that not focusing on the PMF metrics results in poor 
performance on those metrics, doing so may negatively affect their chances of 
expansion or charter renewal. Most of the actions around the PMF are aimed at 
the lowest-performing Tier 3 schools and some low-performing Tier 2 schools, 
while Tier 1 and higher-level Tier 2 schools receive fewer on-site visits and may 


apply for expansion. 


The overall number of PMF metrics and their weights are relatively straightforward. 
The method for schools to earn points in the system based on each metric is 
somewhat more complicated, but it provides context against which school leaders 
can benchmark their performance.®? Each metric has a floor and a target. The floor 
is the minimum value for a metric that will earn any points for a school. The target 
is the maximum value for the metric that will earn the school full points. Schools 
between the floor and the target will earn incrementally more points. Performance 
above the target does not earn any additional points. For example, the floor for 
median growth percentile in grades 4-8 math is 30, and the target is 70. This 
metric is worth 20 possible points. Schools start earning points at 30, and points 
earned rise between 30 and 70. DC PCSB chose this method because it allows 
them to combine metrics measured on different scales and check scoring ranges 
against the distribution of school performance. 


While some floors and targets remain static, DC PCSB recalculates many floors 
and targets on a biannual basis, which includes data analysis, research, and 
stakeholder engagement. This was previously an annual process, but the frequent 
resetting meant that schools did not always have a clear sense of how their goals 
for the year ahead would translate on the PMF.*%* 


DC PCSB uses a “task force” structure to engage school leaders in PMF design 
decisions. Task forces informed the development of specialized PMFs, including 
the early childhood PMF and the adult education PMF. Today, the DC PCSB 
continues to convene regular task force meetings to engage school leaders. Any 
affected school leader may attend, discuss, and vote on proposed changes to 
the PMF for that group of schools. Agendas, meeting notes, and voting results 
are all publicly posted.®° Ultimately these task forces are advisory bodies, but 
feedback from schools is taken seriously in the design process. This lesson was 
hard-earned. One of the earliest iterations of the PMF was created without much 
input and engagement from schools; that version was never implemented and 
heavily revised following school leader pushback, as the board realized it could 


not effectively incentivize school improvement without buy-in on the framework.°° 
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Family and Community Information 


Families may look to the PMF tier ratings as an indicator of school quality, but the 
system is not primarily designed to communicate with parents and community 


members. 


At a high level, the three-tiered system of the PMF is easy to understand, and is 
presented to families and community members in multiple lists and formats on 
the DC PCSB website. Families who want a deeper look at the data can consult 
individual school quality reports available on DC PCSB’s website. School quality 
reports are only available in English, with an array of points, floors, targets, and 
percentages with minimal definitions and context. The school quality reports 
refer to the PMF technical guide for more information, an even denser document. 
PMF results are linked alongside qualitative site reviews and state equity reports, 
without explanation of the differences between these data sources. To provide an 
alternative view of PMF information, DC PCSB has created and released a parent 
guide in multiple languages that highlights the tier and other relevant information 
about each school to assist families in the school choice process. Additionally, a 
high-level PMF overview document, with an explanation of the tiers, metrics, and 
purpose behind the PMF, is available on the DC PCSB website.°” 


Other data sources for information on D.C. schools are meant to be more parent- 
friendly. DC PCSB played a lead role in the creation of MySchoolDC, the district’s 
unified enrollment system for families to compare and find schools, but the PMF 
plays a minor role in school profiles on the site. PMF reports now have an out-of- 
the-way link at the bottom of a school’s profile. New ESSA-aligned STAR ratings, 
which apply across charter and district schools, are prominently displayed in 
search results and at the top of each profile.°® Families may also be confused 

by differences between the PCSB PMF Tiers and the new STAR ratings, which 
were created and implemented by the D.C. Office of the State Superintendent of 
Education. It is possible that as STAR ratings become more established, the PMF 


will become less of a parent- and community-facing system. 
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The New York City (NYC) public school system is the nation’s largest school 
district, enrolling over 11 million students in 1,800 schools.8° NYC school quality 
reports are controlled by the New York City Department of Education (NYC DOE), 
and apply to all schools in the system, including charter schools authorized by 
the district, the state, and the State University of New York. NYC is a mayorally 


controlled school district, and the mayor appoints a chancellor. 


From 2007 to 2013 NYC had a local SPF created by former Mayor Mike Bloomberg 
and Chancellor Joel Klein. This framework assigned an A to F overall grade to 
each school. Beginning in 2014, changes to the system brought together school 
results and progress data with results from on-site quality reviews, increased 

the weight of student survey results, eliminated overall grades, and introduced a 


family-friendly version of the report. 


Since 2014 in its current form. 
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The primary purpose of the NYC School Quality Reports currently is to provide 
school leaders and district superintendents with consistent data about student 
and school performance to drive school improvement.® In recent years, the 
school quality reports were used to identify schools in need of targeted support 
for the renewal schools turnaround program, but the department announced the 
end of the program in 2019.°' As the system has evolved, parents and families 


have become another audience for the school quality reports. 


The NYC DOE publishes three online school quality reports:° 


e School Quality Snapshot: shorter report designed for families and the general 


public that highlights key aspects of school performance 


e School Quality Guide: detailed report that is designed for school leaders to 


identify areas for school improvement 


e School Quality Dashboard: multiyear, detailed school data that is designed to 


be densely packed so that multiple years of data can be viewed on one page 


NYC School Quality Reports do not have summative ratings for schools. They 

rate schools on a four-point scale of Excellent to Needs Improvement on seven 
elements. These elements correspond to the NYC DOE’s “Framework for Great 
Schools,” which was adopted at the same time the quality reports shifted.°? There 
are a large number of metrics in NYC’s system. For example, there are 20 metrics 
that go into a student achievement rating for an elementary school. Metrics vary 
by grade level, including early childhood, elementary/middle school, and high 
school iterations. The primary student achievement data sources for elementary/ 
middle school ratings are growth and performance on state tests, including gap 
closure. Metrics for high school student achievement include graduation, college 
and career readiness, progress toward graduation, performance on state end-of- 
course exams, and closing graduation achievement gaps. Additional qualitative 
ratings are based on periodic structured on-site quality reviews and annual school 
survey results.°* Because of the volume and variation in metrics, individual metrics 


contribute to the overall rating in each element category in different ways. 


Element 


Description and Source of Rating 


Student 
Achievement 


This rating looks at students' performance and growth in reading, math, and high 
school end-of-course exams, including gap closure, as well as graduation rates, 
graduation gaps, college and career readiness, and postsecondary enrollment. 


Rigorous Instruction 


This rating looks at how well curriculum and instruction are designed to engage 
students and foster critical thinking skills, and are aligned with state standards. This 
rating combines results from the Quality Review and NYC School Survey. 


Collaborative 
Teachers 


Supportive 
Environment 


This rating looks at how well teachers participate in opportunities to develop, grow, and 
contribute to the continuous improvement of the school community. This section rating 
combines results from the Quality Review and NYC School Survey. 


This rating looks at how well the school establishes a culture where students feel 
safe, challenged to grow, and supported to meet high expectations. This rating 
takes into account Quality Review, the NYC School Survey, and moving students 
with special needs to less restrictive environments, as well as chronic absenteeism. 


Effective School 
Leadership 


This rating looks at how well school leadership inspires the school community with 


a clear instructional vision and effectively distributes leadership to realize this vision. 


This rating combines results from the Quality Review and NYC School Survey. 


Strong Family- 
Community Ties 


This rating looks at how well the school forms effective partnerships with families 
and outside organizations to improve the school. This rating combines results from 
the Quality Review and NYC School Survey. 


Trust 


This rating looks at whether relationships between school leaders, teachers, 
students, and families are based on trust and respect. The rating is based on NYC 
School Survey responses. 
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System Management and Accountability 


When the NYC DOE first created a local SPF in 2005 under the administration of 
Mayor Mike Bloomberg and Chancellor Joel Klein, the SPF had a clear system 
accountability purpose. Schools were graded on an A to F scale, primarily based 
on state assessment results. Schools with low ratings were subject to increasing 
interventions, the most severe of which was closure. At the same time, a subset 
of schools was selected into an “Autonomy Zone,” where principals had more 
building-level autonomy as long as they met performance targets aligned to the 
SPF.v" 


Beginning in 2014, Mayor Bill de Blasio and his first Chancellor Carmen Farifia 
brought a much different perspective to the school quality reports, and made 
changes which they believed would offer greater depth of information about 
school quality. Schools no longer receive summative grades, and redesigned 
quality reports merge together academic achievement results like test scores 
and graduation with more qualitative results of student and teacher surveys and 
periodic on-site instructional quality reviews. On-site reviews existed prior to the 


SPF, but they had previously been handled separately from the school grades.®° 


The data on the reports are detailed and disaggregated, and the sub-scores for 
each of the seven elements give leaders common information and a framework 
to differentiate schools. Academic achievement metrics are the most detailed 
and have the most emphasis in the framework. The six other elements within 
the framework draw mostly from just two data sources: school quality reviews, 
which are not conducted annually in every school, and student surveys. As a 
result, the reports likely do not provide a full and reliable picture of some of the 
more complex constructs the framework aims to measure, such as strong family- 


community ties. 


At the moment it is unclear how the quality reports factor into system decisions 
such as closure, intervention, or improvement strategies. For example, school 
quality reports played a role in the selection of low-performing schools for the 
“Renewal Schools” program, a school turnaround effort that has since been 
disbanded. It is possible that NYC’s superintendents, who manage 46 subdistricts 
of schools within the NYC DOE,* use the reports differently in activities such as 


principal management or school oversight. 


¢ School Continuous Improvement 


In their current form, NYC’s school quality reports primarily support school 
continuous improvement. When new reports were first released, district materials 
described the overall grades as “oversimplified and inaccurate,” and said new 
reports would “allow schools to identify and address specific strengths and 
weaknesses.”© The Framework for Great Schools, which underpins the design 
and metrics in the school quality reports, was inspired by research on essential 
elements of high-performing schools conducted by the Chicago Consortium on 
School Research (the 5Essentials survey, an element in Chicago’s SPF, came out 


of the same body of work).°° 


School quality reports offer principals a large array of quantitative and qualitative 
data, and several ways in which to view and engage with the data.”? For example, 
in the online school quality guides, academic achievement data is graphed against 
other schools in the system on the basis of performance and impact (growth). 
Academic achievement metrics are compared against demographically similar 
schools, the city average, and the district average. Principals can also compare 
their qualitative performance on survey and site visit measures against city and 
district averages, and look at performance trends over time. These reports are 
available publicly through the NYC DOE online “Infohub,” and are linked in each 


school site. 


Data on the school quality reports are in-depth, comprehensive, and 
disaggregated, providing a wealth of data that a principal might find useful. But 
there could be greater differentiation between diagnostic indicators and outcomes 
than the Framework for Great Schools and reports provide. Principals (and other 
district staff) must decide how to prioritize and use the data they see in the 

quality reports for each school. Additional resources on “Applying the Framework 
for Great Schools” aim to help school leaders navigate the complexity of the 
framework, and translate it into action plans for their school.” This involves deep 
study of the elements of the framework, a needs assessment and goal-setting 
process, and ongoing cycles of planning, implementation, and evaluation aligned 


to the framework. 
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Although the framework is meant to inform continuous improvement, most metrics 
are available annually, which limits the time around which schools can take action. 
Some metrics, such as the results of on-site reviews, are assessed on a multiyear 


basis, which does not allow for timely feedback and improvement. 


F amily and Community Information 


NYC has made progress in communicating its new SPF to parents, but there is still 
a long way to go. School quality reports are now available online, in translatable, 
dynamic, and cellphone-friendly formats, which is a step forward from PDF reports 
in the past. On the parent- and community-facing website schools.nyc.gov, quality 
reports are described as “an overall picture of quality for my school.” Staff from the 
school quality team have used focus groups of parents to inform redesign of the 
reports and make them more accessible. 


But even the most simplified version of the reports is complex. There are eight 
tabs of information, separate quality reports for PreK grades and other grade 
levels in the same school, and no explanation of how individual metrics contribute 
to the ratings schools receive in each subarea. Indications of whether a school is 
improving or declining in performance is available in a different report that is less 
parent friendly. 


Other information parents may want, such as school hours, transit, etc., are in the 
NYC DOE’s school finder, schools.nyc.gov/find-a-school, which links to the school 
quality guide. However, parents going to the NYC DOE website looking for school 
enrollment information would not see any links to the school quality reports or 

snapshots, which are housed in a different part of the website and not referenced 


in the enrollment information.’ 
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Performance Scores 


Approximately 49,000 students attend 85 public schools in New Orleans.”? Nearly 
all of these schools are public charter schools, making New Orleans unique in the 
nation. Schools in New Orleans are overseen by the Orleans Parish School Board, 


and the district as a whole is now known as NOLA Public Schools. 


From 2005-2017, the state-run Recovery School District (RSD) authorized 

and operated most schools in New Orleans in the period of time following 
Hurricane Katrina. During that time, the state rating system played a critical 
local role in evaluating school performance, and the state school performance 
scores operated much like a local SPF. Control over schools transitioned back 
to the locally elected school board beginning in 2017, but NOLA Public Schools 
continues to use the state performance scores and grades as a key piece of its 


school accountability approach.” 


Louisiana has issued school performance scores since 1999; letter grades 
began in 2011. Recent major revisions occurred in 2013 and 2018, with the 


passage of ESSA. 
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School performance scores are part of Louisiana’s plan to identify and intervene 
in low-performing schools under ESSA.”° All traditional public schools, charter 
schools, and publicly funded early learning providers receive ratings. However, 
state-level school performance scores predate ESSA. Scores are used to 
communicate school performance to school leaders and incentivize changes in 
behavior that improve student outcomes,” communicate school performance with 
families and the public,” and inform local decisions. The Orleans Parish School 
Board (OPSB), which oversees NOLA Public Schools, uses school performance 


scores as a key factor in the charter school oversight and renewal processes. 


School performance scores are reported and shared with the public through 
the Louisiana School Finder interactive website.” The Louisiana School Finder 
also provides program data and information about subgroup performance, 
the teacher workforce, and school discipline and attendance, but these data 
are not factored into a school’s performance score. LDOE and NOLA Public 
Schools also offer detailed, downloadable tables of school performance 
scores on their respective websites. Schools rated D or F by the state receive 
more intensive annual monitoring, and F schools are not eligible for a charter 
renewal. Schools with higher letter grades are eligible for extended charter 


terms of up to 10 years before another renewal cycle.”? 


Louisiana’s rating system has very few metrics compared to other systems in this 
report, and solely assesses student outcomes without additional school quality 
or environment measures. This is intentional, to align with the states’ goals 

and emphasis on meeting high standards. Schools accumulate points based 

on student academic performance on each metric in the system. Schools earn 

a score and a letter grade equivalent for each key component of the school’s 
performance rating. Points are weighted and combined to determine a school’s 


overall school performance score and A-F letter grade. 


The state chose to use an assessment performance index, which includes all 
students’ scores, rather than a simple proficiency or mastery rate. The index 
credits schools for student improvement across achievement levels, not just the 


overall percentage of students above the proficiency line. 


Elementary Schools Middle Schools High Schools 
Metric Weight Metric Weight Weight 
State assessment 75% State assessment 70% Mastery of key 25% 
performance (ELA, oerformance (ELA, skills (ELA, algebra, 
math, science, social math, science, social geometry, US history, 
studies), as measured studies), as measured biology) + ELA & 
by a performance index by a performance math growth 
index 
Student assessment 25% Student assessment 25% ACT/Workkeys 25% 
progress progress 
Credit accumulation 5% 4-year graduation 25% 
rate 
Strength of diploma 25% 
index®° 


Val 


System Management and Accountability 


The Louisiana school performance scores support several system management 
and accountability purposes at the state and local level. According to Jill 
Zimmerman, director of accountability policy at the Louisiana State Department 

of Education, ratings “should accurately reflect the quality of schools and 

charter operators, so that districts can make decisions around school expansion, 
replication, closure, and renewal.”*' Within the state’s ESSA plan, schools receiving 
aD or F score for three consecutive years or that have graduation rates below 
67% are identified for comprehensive intervention. The state’s accountability 
system incorporates other factors beyond the school performance scores, notably 
identification of targeted improvement schools based on subgroup achievement, 
and the possibility of state intervention via the Recovery School District. However, 


this analysis focuses narrowly on the SPF itself. 
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When the state-controlled Recovery School District oversaw most schools in New 
Orleans, it made decisions to open, expand, and close schools based on these 
scores. NOLA Public Schools continues to use state scores in a similar way — 
offering highly rated schools more growth potential, and intervening in low-rated 
schools. Schools with three consecutive D ratings may have their charter revoked; 
schools with F ratings are ineligible for charter renewal and may have their charter 


revoked early. 


During the transition back to local control, NOLA Public Schools evaluated 
whether and how to use the state SPF. In community conversations, NOLA Public 
Schools leaders found that many community members understood the SPF results 
and found them credible. “Many stakeholders were already deeply familiar with 
and invested in the state letter grades,” said Dina Hasiotis, senior school support 
and improvement officer for OPSB/NOLA Public Schools, and retaining them 
created a sense of stability during a big governance transition.®* Using the state 
system gives the district less control over the data and metrics. For example, as 
the state changes its grading business rules this year, the local system may need 
to adjust. But, says Hasiotis, “the state has a lot of expertise and capacity when it 
comes to working with the data and ensuring rigor and reliability in the SPF,” and 
therefore NOLA Public Schools does not need to invest as much in building this 


capacity internally. 


The Louisiana system is intentionally simple, with an A to F rating, elementary 

and middle schools focused primarily on state test performance across four 
subject areas, and a smaller weighting toward student growth. The addition of a 
student growth component in 2013 was an important development for the SPF. 

At the high school level there are four equally weighted categories: state test 
performance/growth, college entrance exams, graduation rates, and a strength of 
diploma index, which is based on the share of students graduating with additional 
credentials such as AP/IB credits or early college course completion. State leaders 
in Louisiana have decided that a relatively simple system with high standards for 
top scores is more likely to drive action from educators and from system leaders. 
An SPF ata state level also has different considerations than a local SPF — it must 
meet the requirements of federal law, and function across a wide variety of school 


and district contexts. 


Among the five SPFs we examined, this is the only one weighting performance 
more heavily than growth. It also does not include any specific measures focused 
on student subgroup performance or achievement gaps, although the state does 
measure subgroup gaps in other aspects of its accountability system outside of 
the SPF. This design is intended to focus school leaders on clear goals that should 
motivate them to make changes in instructional programs where needed, but 
system leaders recognize that the scores themselves are not designed to provide 
a roadmap for what changes should be made. This focus on clear “goalposts” 
aligns with one of the Louisiana Department of Education’s guiding beliefs: 
“Louisiana has worked hard to raise expectations for students, and as a result, 


students are performing at higher levels than ever before.” 


The system’s 25% growth measure reflects a compromise between state leaders, 
who perceive a growth measure as less motivating relative to state goals for 
proficiency for all students, and charter advocates, who sought a 50/50 growth/ 


proficiency weight in the system. 


In an SPF heavily weighted toward achievement, students’ incoming skills might 
drive scores. A system like this could disadvantage schools serving students 

who enter below grade level, and put schools at risk of closure based on their 
population rather than the quality of their instruction. At the state level, this risk is 
mitigated by using a performance index rather than a proficiency percentage. A 
performance index allows schools to improve their scores as students move closer 
to proficiency over time, encouraging a focus on students across the performance 


spectrum. 


Additionally, NOLA Public Schools weighs growth more heavily in charter 
revocation and renewal decisions — a D or F rating initiates a process that involves 
more review, site visits, and data collection, and that may lead to revocation. 
Importantly, OPSB uses two years of school performance data when making 
charter revocation and renewal decisions to bring stability to the process. Because 
many New Orleans schools fall into the C or D categories in the state SPF, OPSB 
considers growth rates as a way to differentiate among the low-performing, but not 
F-rated schools. Within D-rated schools, OPSB breaks performance into quartiles 
based on growth metrics; D-rated schools in the top quartile for growth are treated 


differently than D-rated schools with low growth rates. 
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Louisiana’s school performance scores are not meant to diagnose schools’ 
challenges, and it does not measure any leading indicators or environmental 
measures of success. The state’s expectation is that schools use their scores 
to set goals and focus on rigorous and effective instruction. “We expect school 
leaders should focus on what the scores are measuring, which is primarily 


mastery, and work on getting their students there,” said Zimmerman. 


Similarly, as an all-charter school district, NOLA Public Schools is focused on 
authorizer-level actions. System-wide continuous improvement is a district 
priority, but school improvement plans and day-to-day decisions are made 

and implemented by school operators. If schools appear to be on a downward 
trajectory in the SPF, or consistently score poorly, local district leaders will meet 
with the school leadership to discuss improvement plans and provide additional 
expertise in some cases. According to Hasiotis, “we are continuing to explore 


what our support function looks like as a nontraditional district.” 


Louisiana also thinks about another user group for its SPF that other systems did 
not explicitly emphasize: educators. Zimmerman described communicating with 
educators as the state’s top priority for their system, particularly communicating 
the importance of rigorous instruction aligned to state standards. In this way, 
school performance scores may be operating as a different kind of school 
improvement tool that focuses on motivating classroom teachers more than 


principals. 


[70] Bellwether Education Partners 


Family and Community Information 


The Louisiana Department of Education is heavily invested in communicating 
school data to parents. LouisianaSchools.com is a recently redesigned parent- 
facing website that allows parents to search for schools, see performance scores, 
access key programmatic information and deeper data, and make comparisons 
between schools. This same site also includes ratings and profiles for all publicly 
funded early childhood education programs. A review by the Data Quality 


Campaign describes Louisiana’s approach as a “one-stop shop.”™4 


The parent-facing report card website includes data that goes beyond the scores, 
including subgroup data, teacher data, and discipline/attendance data. Many of 
these data elements are federally required under ESSA, but it is interesting that 


parents are shown a wider array of data than schools are graded on. 


School performance scores are also integrated into EnrollNOLA, the unified 
enrollment system for New Orleans. School profiles on EnrollINOLA include 
the grade and performance score for each school, along with when it is up for 


renewal, and a link to the school’s profile on LouisianaSchools.com. 


NOLA Public Schools publishes a parent guide with SPF grades and additional 
data, and a local school performance profile will soon be available to the public. 
This profile will include school environment and student experience information 


that the state SPF does not capture. 


Even though New Orleans is one of the most parent-choice-centered school 
districts in the country, and parent-facing resources are exemplary in some 
ways, there is still progress to be made in communicating and translating 

school information for parents. For example, EdNavigator, a New Orleans-based 
organization that partners with businesses to guide parents through their school 
choices, has found deep demand for help interpreting information about schools 
as it relates to their child. 
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Performance Framework (SPF) 


Over 92,000 students attend 207 schools in Denver, including district-managed 


schools, innovation schools, and charter schools authorized by DPS. 


The Denver SPF launched in 2008. The superintendent and school board 
intended to use the SPF results to measure and understand the performance of 
all schools in the district’s portfolio consistently, including traditional, charter, and 
innovation schools. The SPF is not required by state law and is separate from the 
state ESSA system. 


The Denver SPF has been in use since 2008.°° 


DPS uses its SPF to monitor school progress, support struggling schools, and hold 
schools and district leaders accountable for school performance. The SPF applies 
to all district-run schools, innovation schools (district-run schools with increased 
flexibility and autonomy), charter schools, and alternative schools. Additionally, 
the SPF is a tool that helps students and families understand how their school is 


performing, and families may use SPF ratings during the school choice process.®° 


Communication 


Metrics, Weights, 
and Ratings 


Each year, DPS releases new SPF ratings on its website in a static format. The 
district publishes a two-page version of the school rating called the School 
Summary Report, which includes a rating for each of the six categories in the 
SPF as well as a longer, more detailed report called the School Detail Report that 
explains the points earned on each metric in greater depth.®”? School profiles on 


DPS’ school finder also incorporate high-level results from the SPF. 


Denver uses a color-coded rating system to show how schools are performing. 
The points accumulation and rating assignment methodology is not clearly 
explained on the DPS website, and according to interviews with former district 
officials, methodology can change substantially year to year. Depending on 
available data, different schools within the same grade span have different 
possible point totals in each of the indicator subcategories, and in the total ratings. 
The percentage of points a school earns out of the points possible on each 
indicator determines a rating: Distinguished (blue), Meets Expectations (green), 
Accredited on Watch (yellow), Accredited on Priority Watch (orange), Accredited 


on Probation (red). 


Schools are evaluated on the following indicators; metrics vary by grade level 


served, and some “bonus” points for specialized program offerings are possible: 


Indicators 


Student Progress — Growth (state assessments, early literacy assessments, 
English learner assessments) 


Student Achievement — Status (state assessments) 


Family and Student Engagement and Satisfaction (based on schools’ 
annual surveys) 


High School Postsecondary Readiness — Status (graduation and 
remediation rates) 


High School Postsecondary Readiness — Growth (improvement on 
graduation and remediation rates) 


Academic Gaps (performance and growth of historically underserved student 
groups including English language learners, students with disabilities, students 
in poverty, students of color) 
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System Management and Accountability 


The SPF was primarily designed to serve system management and accountability 
purposes. Schools rated highly in the SPF are rewarded financially with an 
additional $100 per pupil. Schools that receive low ratings get additional 
monitoring and intervention, and may be at risk of restart. The SPF is considered 
as part of innovation school applications and renewals. Denver offers a common 
school performance framework that applies across all schools, including charter 
schools. For charter schools, performance on the SPF is a significant factor 

in charter renewal and expansion decisions. According to Jennifer Holladay, 
director of school development and support at DPS, “a large percentage of the 
district’s decision to renew a charter or close a school is based on the school’s 


performance on the SPF.”®° 


Over the course of more than 10 years, the SPF has become closely enmeshed 
with many of DPS’ oversight and accountability processes. The inner workings 
and business rules of the Denver system are not publicly available, making it 
difficult to understand the relationship between ratings, school governance 


actions, metrics, and methods. 


The SPF awards points to schools based on various metrics. The points are 

added to determine the school’s rating in one of five color categories: Blue 
(Distinguished), Green (Meets Expectations), Yellow (Accredited on Watch), Orange 
(Accredited on Priority Watch), and Red (Accredited on Probation). DPS officials 
use SPF performance and ratings to determine the level of central office support 
and intervention each school receives. DPS uses a Tiered Support Framework to 


target resources and assistance to struggling schools. 


DPS sorts metrics into six indicator categories, with an emphasis on academic 
growth. Like many districts, Denver has adjusted and added metrics to its system 
over time, but some recent additions may have negatively impacted the reliability 
and credibility of the SPF for system management and accountability purposes. 
In 2017, DPS added a new metric measuring early literacy gains. This metric was 
in line with research around the importance of early literacy, but it was not fully 
aligned to the rigor or reliability of state assessments. That year, a record number 


of Denver schools earned the district’s top ratings: blue and green.°° District 


leaders publicly acknowledged that methods used to calculate school scores 
resulted in overstating literacy gains and inflated academic performance scores, 
but DPS did not issue corrected SPF data.*" 


The “academic gaps indicator” has also been the subject of recent controversy. 

This indicator, introduced in 2016, aims to shine a light on educational disparities by 
measuring how traditionally underserved students scored on tests compared to set 
benchmarks and how those results compare to students not identified as traditionally 
underserved. When DPS introduced gaps indicators, it implemented a new policy in 
which schools scoring poorly on this indicator could not earn the district’s top SPF 
scores. In response, teachers, parents, and community leaders have argued that the 
indicator is merely a measure of school demographics, and that it disadvantages 


schools with large populations of traditionally underserved students.°? 


School Continuous Improvement 


The SPF is not designed to serve as a day-to-day continuous improvement tool 
for schools, but the district does use the SPF to communicate priorities to school 
leaders through its metrics. As mentioned earlier, many school building leaders 
have a high degree of autonomy in Denver and this autonomy applies to the SPF 
as well. School leaders can use the data in the SPF to inform their day-to-day 
school planning and instruction, or not. Most of the actions tied to the SPF are 
aimed at schools with lower ratings in danger of closure or restart. Schools with 
higher ratings have less incentive to aim for continuous improvement, absent a 


desire for expansion. 


One metric the district uses to drive continuous improvement, particularly among 
top-performing schools, is the “Academic Gaps” indicator. Schools that perform 
poorly on this indicator cannot receive the SPF’s top ratings. According to public 
advocacy organization At+ Colorado, in 2018, “20 schools that earned sufficient 
points overall to be Green had their rating lowered to Yellow because they 

did not meet expectations on the Academic Gaps indicator.”*? By placing this 
extra emphasis on the achievement gap between student groups, DPS sends a 
clear message to school leaders that they should focus on closing gaps among 


student subgroups. 
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Differences between the state and district systems may create confusion among 
school leaders about which student groups to focus on and which student 
outcomes matter the most. The district excludes students who leave or join a 
school midyear from its performance data, because of high within-district student 
mobility. The state data includes all students who take the state test regardless of 
when they become a student at the school, while the district does not. As a result, 
the district and state publish different sets of school assessment performance. 


There are no current plans to align the SPF assessment metrics with the state. 


F amily and Community Information 


The SPF was always intended to serve as an informational tool for families as 
well as district and school leaders. Changes over time have helped bring the SPF 
closer to this goal. Early advocacy from parent groups led the district to make 
significant changes to the data and ratings so families could understand them 
more easily, including the introduction of color-coded ratings that are core to the 


system today. 


Since then, district leaders have made attempts to improve the readability and 
design of public-facing SPF information. For parents, the district provides a School 
Summary Report with information about the school’s rating. The overall rating is at 
the top of the profile alongside trend data over time. For each of the six indicator 
categories in the SPF, DPS provides a color-coded sub-rating and a summary of 
the points earned in each category. The district places an additional emphasis on 
equity by including a breakdown of how each school performs for four historically 


underserved groups. 


The SPF is available in a total of 10 languages, so that parents reading ina 
language other than English also have access to school quality information. DPS 


also offers information about the SPF in explainer videos. 


As in other districts, district leaders in Denver struggle to balance parent priorities 
around school information and systemic use considerations. Denver parents 

have requested more information about school culture, diversity, and other 
environmental metrics. But adding these metrics may further complicate an 
already complicated system, and stray from the focus on student outcomes that 


defines DPS’ accountability system for schools. 


District leaders have additional work left to make the SPF more transparent to 
families and community leaders. DPS does not provide easily available public 
information about the methods used to translate data into school tiers or 
determine points earned on each indicator, and changes in the system over time. 
Recent controversies, like the destabilizing effect of adding the early literacy 
indicator, have only strengthened calls for the SPF to be more transparent about 


how scores are calculated, and why and how those calculations shift over time. 
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